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Joint  Stars  Full-Rate  Production 
April  25, 1997  Decision  was  Premature  and  Risky 

Congressional  Committees 

We  have  reviewed  the  Department  of  Defense’s  (dod)  recent  decision  to 
commit  to  the  fuU-rate  production  of  the  Joint  Surveillance  Target  Attack 
Radar  System  (Joint  stars).  More  specifically,  we  analyzed  whether  (1)  the 
system  had  demonstrated  a  level  of  maturity  through  testing  to  justify  a 
fuU-rate  production  commitment,  (2)  dod  considered  and  resolved 
important  cost  and  performance  issues  prior  to  making  its  decision,  and 
(3)  there  are  future  actions  that  could  reduce  program  risk.  This  review 
was  performed  under  our  basic  legislative  responsibility  and  we  are 
addressing  it  to  you  because  it  falls  under  your  committees’  jurisdiction. 


Background 


Joint  STARS  is  a  joint  Air  Force  and  Army  wide-area  surveillance  and  target 
attack  radar  system  designed  to  detect,  track,  classify,  and  support  the 
attack  of  moving  and  stationary  ground  targets.  This  $11  billion  major 
defense  acquisition  program  consists  of  air  and  ground  segments — 
refurbished  707  aircraft  (designated  the  E-8)  equipped  with  radar, 
operation  and  control,  data  processing,  and  communications  subsystems, 
together  with  ground  stations  equipped  with  communications  and  data 
processing  subsystems. 

Low-rate  initial  production  (lrip)^  of  the  Joint  stars  aircraft  began  in  fiscal 
year  1993.  In  line  with  10  U.S.C.  2399,  dod’s  final  decision  to  proceed 
beyond  lrip  first  required  the  dod  Director  of  Operational  Test  and 
Evaluation  (dot&e)  to  submit  a  report  to  Congress,  referred  to  as  the 
Beyond  lrip  report,  stating  whether  (1)  the  test  and  evaluation  performed 
was  adequate  and  (2)  testing  demonstrated  that  the  system  is  effective  and 
suitable  for  combat,  that  is,  operationally  effective  and  suitable.^ 

'The  Joint  stars  aircraft  was  scheduled  to  begin  its  initial  operational  test 
and  evaluation — referred  to  as  the  Joint  stars  multi-service  operational 
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*Low-rate  initial  production  of  systems  is  to  produce  the  minimum  quantity  necessary  to  (1)  provide 
production-configured  or  representative  articles  for  operational  test  and  evaluation,  (2)  establish  an 
initial  production  base  for  the  system,  and  (3)  permit  an  orderly  increase  in  the  production  rate  for  the 
system  sufficient  to  lead  to  full-rate  production  upon  the  successful  completion  of  operational  test  and 
evaluation. 

“Operational  effectiveness  refers  to  the  ability  of  a  system  to  accomplish  its  mission  in  the  planned 
operational  environment.  Operational  suitability  is  tbe  degree  to  which  a  system  can  be  placed 
satisfactorily  in  field  use  considering  such  factors  as  reliability  and  maintainability. 
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test  and  evaluation® — ^in  November  1995.  That  testing  was  delayed  and 
then  changed  because  of  the  deployment  of  Joint  stars  assets  to  the 
European  theater  to  support  Operation  Joint  Endeavor  in  Bosnia.  The  Air 
Force  Operational  Test  and  Evaluation  Center  (afotec)  and  the  U.S.  Army 
Operational  Test  and  Evaluation  Command  conducted  a  combined 
development  and  operational  test  of  Joint  stars  from  July  through 
September  1995  and  an  operational  evaluation  of  the  system  during 
Operation  Joint  Endeavor  from  January  through  March  1996. 

Two  Air  Force  Joint  stars  aircraft  and  13  Army  Joint  stars  ground  station 
modules  were  deployed  to  support  Operation  Joint  Endeavor  and 
operationaEy  evaluated  from  January  through  March  1996.  After  analyzing 
the  data  from  the  combined  development  and  operational  test  and  the 
operational  evaluation  performed  during  Operation  Joint  Endeavor, 
AFOTEC  issued  its  Joint  stars  multi-service  operational  test  and  evaluation 
final  report  on  Jime  14, 1996.  dot&e  staff  analyzed  the  same  and  additional 
data  and  the  Director  issued  his  Beyond  lrip  report  to  Congress  on 
September  20, 1996.^  On  September  25, 1996,  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  signed  an  acquisition  decision 
memorandum  approving  the  Joint  stars  program’s  entry  into  full-rate 
production  with  a  total  planned  quantity  of  19  aircraft. 


Results  in  Brief 


The  Joint  Surveillance  Target  Attack  Radar  System’s  performance  during 
its  combined  development  and  operational  test  and  the  operational 
evaluation  done  in  Bosnia  do  not  support  a  decision  to  commit  the  system 
to  full-rate  production.  The  system’s  operational  effectiveness  and 
suitability  were  not  demonstrated  during  the  operational  testing.  For 
example,  the  dod  Director  of  Operational  Test  and  Evaluation  could  only 
state  that  the  system  had  demonstrated  effectiveness  for  “operations  other 
than  war”  and  foimd  that  the  system  “as  tested  is  unsuitable.”  He  further 
reported  that  only  18  (25  percent)  of  71  performance  criteria  tested  were 
demonstrated  met  by  the  system  and  that  further  testing  is  required  for  the 
remaining  53. 


^The  multi-service  operational  test  and  evaluation  was  to  consist  of  a  combined  development  and 
operational  test  and  a  dedicated  operational  test 

‘‘Our  analysis  focused  principally  on  the  details  in  the  **Beyond  LRIP”  report  rather  than  just  the 
conclusions  in  the  letter  transmitting  that  report  to  Congress.  Those  details  provide  a  clearer  picture  of 
Joint  stars’  performance.  For  example,  while  the  letter  states  “as  far  as  suitability  is  concerned,  [the 
system]  did  not  meet  its  requirements  in  [Operation  Joint  Endeavor],  a  problem  which  would  be 
exacerbated  in  a  higher  intensity  conflict,”  the  report  states  that  Joint  stars  “as  tested  is  unsuitable” 
and  provides  detailed  examples  of  its  suitability  problems. 
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dod’s  decision  to  move  Joint  stars  into  fiill-rate  production  was  premature 
and  raised  the  program’s  level  of  risk.  The  program  could  have  continued 
under  lrip  imtil  operational  effectiveness  and  suitability  for  combat  were 
demonstrated  and  plans  to  address  identified  deficiencies  and  reduce 
program  costs  were  completed.  Instead,  dod  decided  in  favor  of  Joint 
STARS  fuU-rate  production  without  the  benefit  of  that  information.  During 
the  period  that  the  full-rate  production  decision  was  being  considered,  the 
Assistant  to  the  President  for  National  Security  was  promoting  the  sale  of 
the  system  to  the  North  Atlantic  Treaty  Organization  (nato).  In  an 
August  10, 1996,  memorandum  to  the  Secretaries  of  State,  Defense,  and 
Commerce  and  to  the  Chairman  of  the  Joint  Chiefs  of  Staff,  the  Assistant 
to  the  President  for  National  Security  Affairs  stated  that:  “We  have  been 
working  through  various  military,  diplomatic,  and  political  channels  to 
secure  nato  support  for  a  Fall  1996  decision  in  principle  by  the  Conference 
of  Armament  Directors. .  .to  designate  [Joint  stars]  as  NATO’s  common 
system.”  A  dod  official  informed  us  that  in  November  1996,  the  nato 
armament  directors  delayed  their  decision  on  Joint  stars  for  a  year. 

Before  dod  approved  the  full-rate  production  of  Joint  stars,  dot&e 
provided  Congress  with  a  Joint  stars  Beyond  lrip  report,  as  required  by 
law.  The  report  clearly  indicates  that  (1)  further  operational  testing  is 
needed,  (2)  dot&e  could  only  declare  effectiveness  for  operations  other 
than  war,  and  (3)  the  system  was  unsuitable  as  tested.  Having  issued  this 
report,  dot&e  is  under  no  further  obligation  to  report  to  Congress  at  the 
Beyond  lrip  report  level  of  detail  on  the  adequacy  of  the  operational 
testing  or  on  whether  the  system  has  demonstrated  effectiveness  and 
suitability  for  combat.  However,  dod  plans  foUow-on  test  and  evaluation  to 
address  the  deficiencies  identified  during  the  earlier  testing. 

'There  is  an  opportunity  not  currently  under  consideration  that  could 
reduce  the  Joint  stars’s  program  cost  and  result  in  an  improved  system. 
Since  the  Joint  stars  was  approved  for  lrip,  the  procurement  cost 
objective  of  the  Air  Force’s  share  of  the  Joint  stars  has  increased  by  about 
$1  billion.  This  is  primarily  due  to  the  fact  that  it  is  taking  greater  effort 
and  more  resources  to  refurbish  the  25-30  year  old  707  airframes  than 
previously  anticipated.  The  estimated  cost  of  procuring,  refurbishing,  and 
modifying  each  used  707  airframe  to  receive  the  system’s  electronics  is 
now  about  $110  million.  As  early  as  1992,  the  Boeing  Company  proposed 
putting  the  system  on  newer  Boeing  767-200  Extended  Range  aircraft,  but 
this  proposal  was  not  accepted  at  that  time  as  cost-effective.  Given  the 
current  707  airframe  procurement,  refurbishment,  and  modification  cost 
and  a  1996  price  for  a  commercial  version  Boeing  767-200  Extended  Range 
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aircraft  of  between  $82  million  and  $93  million,  it  may  now  be  more 
cost-effective  for  the  Air  Force  to  buy  that  or  some  other  new,  more 
capable  aircraft.  Such  an  aircraft  could  provide  a  longer  life,  greater  room 
for  growth,  greater  flight  range,  greater  fuel  efficiency,  higher  operational 
availability,  and  lower  program  Ufe-cycle  costs. 


Test  Results  Do  Not 
Support  Full-Rate 
Production 


LRiP  of  the  Joint  stars  aircraft  began  in  fiscal  year  1993.  By  statute,  10 
U.S.C.  2399,  the  “Secretary  of  Defense  shall  provide  that  a  major  defense 
acquisition  program  may  not  proceed  beyond  low-rate  initial  production 
until  initial  operational  test  and  evaluation  of  the  program  is  completed.” 

Operational  test  and  evaluation  is  the  primary  means  of  assessing  weapon 
system  performance  in  a  combat-representative  environment.  It  is  defined 
as  the  (1)  field  test,  conducted  under  realistic  combat  conditions,  to 
determine  an  item’s  effectiveness  and  suitability  for  use  in  combat  by 
typical  military  users  and  (2)  evaluation  of  the  results  of  such  a  test.  If 
used  effectively,  operational  test  and  evaluation  is  a  key  internal  control 
measure  to  ensure  that  decisionmakers  have  objective  information 
available  on  a  weapon  system’s  performance,  thereby  minimizing  risks  of 
procuring  costly  and  ineffective  systems. 

Joint  STARS  was  moved  from  low-rate  to  full-rate  production  even  though 
(1)  it  performed  poorly  during  both  the  combined  development  and 
operational  test  and  the  operational  evaluation  in  Bosnia,  (2)  excessive 
contractor  effort  was  needed  to  support  Operation  Joint  Endeavor,  (3)  the 
suitability  and  sustainability  of  the  system  is  questionable  since  it  uses 
refurbished  25-30  year  old  airframes,  and  (4)  operational  software  is 
considered  significantly  munature. 


Test  Results  Were 
Reported  as  Disappointing 


In  dot&e’s  Beyond  lrip  report,  the  dot&e  stated  that  Joint  stars  had  only 
demonstrated  effectiveness  for  operations  other  than  war.  The  report 
indicated  that  of  three  critical  operational  issues^  to  judge  effectiveness, 
only  one  had  been  demonstrated  as  met  “. . .  with  limitations.”  Those 
critical  operational  issues  related  to  (1)  performance  of  the  tactical 
battlefield  surveillance  mission,  that  is,  surveillance — “met  with 


^DOD  regulation  5000.2-R  states  that  “critical  operational  issues  are  the  operational  effectiveness  and 
operational  suitability  issues  (not  parameters,  objectives  or  thresholds)  that  must  be  examined  in 
operational  test  and  evaluation  to  evaluate/assess  the  system’s  capability  to  perform  its  mission.”  It 
also  states  that  “if  every  critical  operational  issue  is  resolved  favorably,  the  system  should  be 
operationally  effective  and  operationally  suitable  when  employed  in  its  intended  environment  by 
typical  users.” 
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limitations”;  (2)  support  of  the  execution  of  attacks  against  detected 
targets,  that  is,  target  attack  support;  and  (3)  the  provision  of  information 
to  support  battlefield  management  and  target  selection,  that  is,  battle 
management.  The  effectiveness  critical  operational  issues  were  judged 
based  on  seven  supporting  measures.  In  its  report  to  Congress,  dot&e 
listed  four  of  those  measures  of  effectiveness  as  “not  met”  dining  the 
system’s  combined  development  and  operational  test  and  did  not  list  any 
as  having  been  demonstrated  during  the  Operation  Joint  Endeavor 
operational  evaluation. 

However,  of  greater  concern,  according  to  dot&e,  is  the  fact  that  the 
system  did  not  meet  its  overall  suitability  requirements  during  Operation 
Joint  Endeavor.  In  his  executive  summary,  the  Director  stated  that  most  of 
dot&e’s  Joint  STARS  concerns  relate  to  operational  suitability.  He  went  on 
to  say  that 

“In  the  current  configuration,  the  [Joint  stars]  aircraft  has  not  demonstrated  the  ability  to 
operate  at  the  required  maximum  altitude;  adequate  tactics,  techniques,  or  procedures  to 
integrate  [Joint  stars]  into  operational  theaters  have  not  been  developed;  [Joint  stars] 
exceeded  the  break  rate  and  failed  the  mission  reliability  rate  during  [Operation  Joint 
Endeavor].  During  [Operation  Joint  Endeavor],  [Joint  stars]  did  not  achieve  the  effective 
time-on-station  requirement.” 


He  concluded  that  without  corrective  actions,  “[Joint  stars]  would  not  be 
suitable  in  higher  intensity  conflict”  and  later  in  the  report  judged  that  the 
system  “as  tested  is  unsuitable.” 

Analysis  of  dot&e’s  Beyond  lrip  report  indicates  that  not  only  did  Joint 
stars  have  disappointing  test  results  but  also  that  extensive  follow-on 
operational  testing  of  Joint  stars  is  needed.  In  its  Beyond  lrip  report, 
DOT&E  presented  a  table  that  reported  its  findings  of  the  combined 
development  and  operational  test  and  Joint  stars  Operation  Joint 
Endeavor  operational  evaluation  and  indicates  where  further  testing  is 
required.  Our  analysis  of  that  table  indicates  that  at  most  only  25  of  71  test 
criteria  could  be  judged  met.  dot&e  considers  18  of  those  25  to  require  no 
further  testing,  that  is,  dot&e  judges  them  clearly  met.  However,  our 
analysis  also  indicates  that  19  test  criteria  were  clearly  not  met  and  that  as 
many  as  26  might  not  have  been  met.  Twenty-seven  of  the  criteria  could 
not  be  determined  in  either  the  combined  development  and  operational 
test  or  the  Operational  Joint  Endeavor  operational  evaluation.  Of  the  71 
Joint  stars  operational  test  and  evaluation  criteria  listed,  dot&e  indicates 
that  53,  or  about  75  percent,  require  further  testing. 
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In  addition  to  the  above,  dot&e  also  noted  that  there  were  several 
operational  features  present  during  Joint  stars  Operation  Joint  Endeavor 
deployment  that  were  essential  to  its  mission  accomplishment  but  were 
not  included  in  the  recent  production  decision.  It  provided  two  specific 
examples — satellite  communications  and  a  deployable  ground  support 
station,  dot&e  believes  these  features  “will  be  a  necessary  part  of  the 
production  decision  to  achieve  a  capable  [Joint  stars]  system.”  It  also 
noted  the  need  for  other  features — amoving  target  indicator  clutter 
suppression,  communications  improvements,  terrain  masking  tools  for 
ground  station  module  operators,  and  linkage  to  operational  theater 
intelligence  networks.  Since  at  least  two  of  the  features  present  during 
Operation  Joint  Endeavor  were  “essential”  to  its  mission  accomplishment 
have  already  been  developed,  and  may  be  needed  “to  achieve  a  capable 
Joint  stars  system,”  those  features  should  also  be  tested  dming  the 
planned  Joint  stars  foUow-on  test  and  evaluation. 


The  degree  of  contractor  involvement  required  during  the  operational 
evaluation  indicates  increased  program  risk  and  makes  the  reported  Joint 
stars  performance  appear  better  than  it  would  have  otherwise.  The 
multi-service  operational  test  and  evaluation  plan,  in  discussing  contractor 
involvement  during  the  testing,  stated: 

“[Multi-service  operational  test  and  evaluation]  must  yield  the  most  credible  and  objective 
results  possible.  All  facets  of  the  test  effort  must  operate  under  the  rules  that  support  total 
objectivity  and  prevents  improper  data  manipulation.” 


Significant  Contractor 
Involvement 


The  test  plan  also  states  that  interim  contractor  support  “wiU  be  Umited  to 
perform  ground  maintenance  only;  no  in-flight  support.”  Regarding  the 
Army’s  ground  station  modules,  it  states  that  “the  Army  maintenance 
concept  does  not  call  for  [contractor  involvement]  at  any  level . . . .” 

However,  during  Operation  Joint  Endeavor  there  was  significant 
contractor  support  of  the  2  aircraft  and  13  ground  station  modules 
deployed.  According  to  the  afotec  report. 


“Approximately  80  contractors  were  deployed  to  support  the  E-8C.  However,  three  or  four 
[contractor]  systems  engineers  flew  on  each  flight  to  ensure  they  could  provide  system 
stability  and  troubleshooting  expertise  during  missions.  Additionally,  three  or  four 
[contractor]  software  engineers  were  on  the  ground  full  time,  researching  and  developing 
fixes  to  software  problems  identified  during  the  deployment.” 
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AFOTEC  also  reported  that  “Each  of  the  [ground  station  modules]  had  one 
contractor  representative  on  site  and  on  call  with  additional  help  available 
as  necessary.  Five  contractor  representatives  remained  at  [Rhein-Main  Air 
Base]  and  functioned  as  a  depot.”  The  afotec  report  stated  that  the  “test 
director  agreed  to  contractor  participation  in  the  [Operational  Evaluation] 
to  a  greater  extent  [than]  permitted  under  US  Public  Law,  Title  10,  Section 
2399.”  [Emphasis  added.] 

When  we  formally  expressed  our  concerns  about  the  significant  contractor 
involvement  in  Operation  Joint  Endeavor,®  dod  did  not  directly 
acknowledge  that  contractors  were  utilized  beyond  the  constraints  of  the 
law  governing  operational  test  and  evaluations.  It  stated  that  “were  this 
solely  an  [initial  operational  test  and  evaluation],  contractors  would  not 
have  been  utilized  beyond  the  constraints  of  10  U.S.C.  §2399,”  and  noted 
that  the  contractors  were  involved  in  the  Joint  stars  operation  to  support 
the  mission.  It  further  stated  that  employing  Joint  stars  in  Operation  Joint 
Endeavor  “allowed  the  system  to  be  operated  and  tested  at  a  greater 
operational  tempo  than  the  system  would  have  undergone  in  traditional 
testing.”  DOD  also  stated  that  “because  of  the  developmental  nature  of  the 
aircraft,  we  needed  to  have  more  contractor  personnel  involved  than  we 
would  otherwise  have  had.” 

It  is  xmderstandable  that  dod  wanted  to  provide  the  best  support  possible 
in  Operation  Joint  Endeavor.  However,  such  significant  contractor  use 
neither  supports  a  conclusion  that  the  system  is  operationally  effective  or 
suitable  for  combat,  nor  is  it  indicative  of  a  level  of  system  maturity  that 
justifies  full-rate  production. 

Joint  STARS  failure  to  meet  its  maintainability  criteria  during  an  operation 
less  demanding  than  combat,  even  with  such  significant  contractor 
involvement  beyond  that  planned  for  in  combat,  also  raises  the  question  of 
the  Air  Force’s  ability  to  develop  a  cost-effective  maintenance  plan  for  the 
system.  This  issue  is  recognized  in  the  Under  Secretary’s  acquisition 
decision  memorandum  approving  Joint  stars  full-rate  production.  In  that 
memorandum,  the  Under  Secretary  called  for  the  Air  Force  to  fuUy 
examine  Joint  stars  affordability,  sustainability,  and  life-cycle  costs, 
including  the  scope  of  contractor  support. 


^Joint  STAES  Production  Decision  (GAO/NSIAD-96-242R,  Sept.  20, 1996). 
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System  Sustainability  and 
Suitability  Is  Questionable 


In  discussing  the  sustainability  of  the  Joint  stars  system,  dot&e  noted  in  its 
Beyond  lrip  report  that  “It  is  not  yet  known  what  the  operational  tempo 
will  be  for  Joint  stars.”  It  concluded  that 


“If  it  is  determined  that  the  system  will  be  operated  at  rates  similar  to  awacs  [Airborne 
Warning  And  Control  System],  it  is  questionable  whether  the  [Joint  stars  aircraft]  can  be 
sustained  over  time.  Airframe  problems  have  already  been  experienced  on  the  existing 
[Joint  STARS  airframes],  including  a  hydraulics  failure  and  a  cracked  strut  in  the  fuselage 
between  the  wings.” 

In  discussing  the  Joint  stars  aircraft  engines,  dot&e  noted  that  they  “are 
1950s  technology  and  may  not  be  reliable”  and  cited  afotec’s  reporting 
that  engine  failures  were  among  the  principal  reasons  that  the  aircraft 
failed  to  meet  the  break  rate  criteria'^  during  Operation  Joint  Endeavor. 

In  discussing  Joint  stars  suitability,  dot&e  also  noted  that  the  limited 
power  of  the  engines  “made  it  difficult  to  reach  the  aircraft’s  normal 
operating  altitude  of  36,000  feet,  much  less  the  42,000  feet  maximum 
altitude  it  is  required  to  reach.”  It  further  reported  that  during  Operation 
Joint  Endeavor,  the  aircraft  required  approximately  11,000  feet  of  runway 
when  taking  off  wdth  140,000  pounds  of  fuel  and  concluded  that  “this  may 
pose  a  significant  challenge  to  operational  commanders  because  the 
[North  Atlantic  Treaty  Organization]  standard  runway  length  is  8,000  feet.” 
It  noted  that  operational  challenges  would  be  great  in  other  theaters  as 
well  and  cited  Korea  as  an  example.  It  reported  that  Joint  stars 

“. . .  would  face  operational  challenges  taking  off  from  five  runways  in  Korea,  each 
approximately  9,000  feet  long.  Operations  out  of  Korea  would  likely  require  taking  off  with 
less  fuel  and  subsequent  aerial  refueling  or  shortening  the  time  on  station.” 

Another  area  of  Joint  stars  suitability  concern  is  the  system’s  growth 
potential,  dot&e  has  reported  that  it  is  not  clear  that  the  remantifactured 
707  platforms  will  be  capable  of  incorporating  all  of  the  planned  upgrades, 
noting  that  the  airframe  limits  the  system’s  growth  potential  both  in  weight 
and  volume.  It  reported  that  as  the  current  mission  equipment  already  fills 
much  of  the  fuselage,  there  is  little  room  for  expansion,  dot&e  also  noted 
that  increasing  the  payload  weight  would  require  longer  takeoff  runways 
or  taking  off  with  less  fuel,  thus  increasing  the  aerial  refueling  requirement 
or  decreasing  mission  duration. 


'Break  rate  is  one  measure  of  system  suitability.  For  Joint  stars  operational  testing  it  was  defined  as 
the  percentage  of  missions  flown  for  a  specific  period  of  time  in  which  a  previously  mission-capable 
essential  subsystem  was  inoperable  once  the  aircraft  landed. 
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Operational  Software 
Rated  Significantly 
Immature 


DOT&E  also  noted  that  the  system’s  current  computers  limited  its  growth 
potential  due  to  their  having  very  little  reserve  processor  time  or  memory. 
It  stated  that  the  Air  Force  requires  that  no  more  than  50  percent  of  central 
processor  unit  cycles  or  memory  be  utilized  by  a  new  system,  dot&e 
reported  that  “None  of  the  E-8C  computer  subsystems  meet  these 
requirements.”  It  provided  an  example  of  the  problem,  stating  that  “the 
memory  reserve  of  the  operator  workstations  still  does  not  meet  the 
requirement,  even  after  being  increased  from  128  megabytes  to 
512  megabytes  just  prior  to  [Operation  Joint  Endeavor].”  This  assessment 
is  another  indicator  of  the  program’s  elevated  risk.  As  dot&e  noted  “Future 
software  enhancements  and  modifications  may  require  significant 
hardware  upgrades. . . .” 


The  AFOTEC  report  specifically  pointed  to  the  lack  of  maturity  in  Joint  stabs 
software.  For  example,  afotec  reported  that 

•  “during  Joint  stars  [multi-service  operational  test  and  evaluation], 
software  deficiencies  were  noted  on  every  E-8C  subsystem,” 

•  the  software  “does  not  adequately  support  [the]  operator  in  executing  the 
mission;”  and 

•  “Joint  STARS  software  does  not  show  the  expected  maturity  trends  of  a 
system  at  the  end  of  development.” 

In  discussing  Joint  stars  software  maturity,  dod  advised  us  that  the  afotec 
report  judged  the  system  overall  operationally  effective  and  suitable. 
Specifically,  in  reference  to  software  problems,  dod  stated  that  “the 
mtijority  of  software  faults  that  occurred  during  Operation  Joint  Endeavor 
were  resolved  while  airborne  in  less  than  10  minutes.”  However,  both 
afotec  and  dot&e  had  some  critical  concerns  regarding  how  Joint  stars 
software  functioned.  For  example,  according  to  afotec,  the  “Joint  stars 
software  is  immature  and  significantly  impedes  the  system’s  rehability  and 
effectiveness,”  and  according  to  dot&e 

•  “Immature  software  was  clearly  a  problem  during  [Operation  Joint 
Endeavor]. . .” 

•  “. .  .the  prime  contractor  had  to  be  called  in  to  assist  and  correct  69 
software-specific  problems  dming  the  41  E-8C  missions  ...  .an  average  of 
1.4  critical  failures  per  flight. . .” 

•  “Communications  control  was  lost  on  69  percent  of  the  flights.” 

•  “The  system  management  and  control  processor  failed  and  had  to  be 
manually  reset  on  half  of  the  flights.” 
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DOD  has  stated  that  the  Air  Force  “plans  several  actions  to  mature  the 
software  and  provide  the  required  support  resources”  and  that  “an  interim 
software  release  in  April  1997  will  correct  some  software  deficiencies 
identified  during  the  operational  evaluation.”  dod  also  noted  that  software 
updates  will  be  loaded  each  year  thereafter  and  that  software  changes  are 
easily  incorporated.  How  easily  these  software  changes  are  incorporated 
remains  to  be  seen  because  much  of  this  software,  according  to  afotec 
and  DOT&E,  is  poorly  documented.  For  example,  afotec  has  reported  that 
there  are  395  deficiency  reports  open  against  the  Joint  stars  program,  318 
of  which  are  software  related,  dot&e  also  stated  that  the  more  than  750,000 
lines  of  Joint  stars  software  code  are  “poorly  documented”  and  later 
commented  that  “Software  problems  with  the  communications  and 
navigation  systems  were  never  fully  corrected,  even  after  extensive  efforts 
by  the  system  contractor.”®  These  facts  in  combination  with  dod’s 
comments  raise  the  serious  question  as  to  which  software  deficiencies  are 
to  be  addressed  in  the  planned  April  software  update. 


Alternative  Aircraft 
Should  Be  Considered 


There  is  an  opportunity  not  currently  under  consideration  that  could 
reduce  the  Joint  stars  program  cost  and  result  in  an  improved  system. 
Since  the  Joint  stars  was  approved  for  lrip,  the  procurement  cost 
objective  of  the  Air  Force’s  share  of  the  Joint  stars  has  increased  by  about 
$1  billion.  Program  costs  escalated  from  approximately  $5.2  billion  to 
approximately  $6.2  billion  in  then-year  dollars.  A  dod  official  informed  us 
that  of  the  $1  billion  cost  growth,  $760  million  is  attributed  to  the 
increased  cost  to  buy,  refurbish,  and  modify  the  used  707  airframes  to 
receive  the  Joint  stars  electronics.  The  remaining  cost  growth  is  attributed 
to  other  support  requirements  and  growth  in  required  spare  parts. 


At  least  as  early  as  1992,  the  Boeing  Company  proposed  putting  Joint  stars 
on  newer  Boeing  767-200  Extended  Range  aircraft,  but  this  proposal  was 
not  accepted  as  cost-effective.  According  to  the  1996  Boeing  price  list,  the 
commercial  version  of  this  aircraft  can  be  bought  for  between  $82  million 
and  $93  million  depending  on  options  chosen  (this  is  flyaway  cost — ^the 
cost  of  a  plane  ready  to  be  flown  in  its  intended  use).  Furthermore,  the 
flyaway  cost  of  a  commercial  Boeing  757,  which  a  Boeing  representative 
informed  us  is  in  many  respects  more  comparable  to  the  707s  being  used, 
is  listed  at  between  $61  million  to  $68  million.  The  actual  cost  of  procuring 
either  of  these  aircraft  could  be  lowered  by  volume  discounts  and  by  the 


^AFOTEC  stated  that  “Safety  of  flight  is  jeopardized  due  to  invalid  navigational  commands  sent  from 
the  FMS-800  [flight  management  system]  and  conflicting  and/or  invalid  navigational  data  displayed  to 
the  flight  crew.” 
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cost  of  the  commercial  amenities  not  required.  On  the  other  hand,  these 
aircraft  would  require  modifications  to  receive  Joint  stars  equipment, 
which  would  raise  their  cost. 

DOD  informed  us  that  the  cost  of  procuring,  refurbishing,  and  modifying  the 
current  707  aircraft  to  receive  Joint  stars  equipment  is  now  estimated  to 
be  about  $110  million  per  airframe.  The  cost  of  procuring  and  preparing 
new  aircraft  might  be  comparable  or  even  less  than  the  current  cost.  In 
addition,  the  Air  Force  would  acquire  a  new  platform  that  could  have 
(1)  greater  room  for  growth  (both  volume  and  weight),  (2)  take  off 
capability  from  a  shorter  runway,  (3)  greater  time-on-station  capability, 

(4)  significantly  improved  fuel  efficiency,  (5)  extended  aircraft  life  over 
the  707  currently  used,  and  (6)  reduced  operational  and  support  cost. 

In  commenting  on  a  draft  of  this  report,  dod  stated  that  it  considered 
alternatives  to  the  current  air  platform,  both  before  lrip  started  and  at  the 
full-rate  production  decision  point.  It  also  stated  that  the  cost  of  moving 
the  Joint  stars  mission  to  an  alternative  platform  would  outweigh  the 
benefits.  We  note,  however,  that  at  a  meeting  with  dod  and  service  officials 
to  discuss  that  draft,  we  asked  about  the  reported  dod  and  service 
analyses.  One  Air  Force  official  stated  that  the  Air  Force’s  platform  choice 
was  not  revisited  prior  to  the  full-rate  production  decision.  None  of  the 
other  13  dod  and  service  officials  present  objected  to  that  statement. 
Furthermore,  when  we  asked  for  copies  of  the  air  platform  analyses  that 
were  done  in  support  of  either  the  low-rate  or  the  full-rate  production 
decision,  dod  was  unable  to  supply  those  analyses.  Finally,  dod  officials 
have  informed  us  that  a  Command,  Control,  Communications,  Computers, 
Intelligence,  Surveillance,  and  Reconnaissance  Mission  Assessment®  has 
been  performed  that  indicates  that  the  Air  Force  could  acquire  a  more 
effective  system  while  saving  $3  billion  through  the  year  2010  by  moving 
the  Joint  stars  mission  to  either  a  business  jet  or  an  unmanned  aerial 
vehicle  following  the  procurement  of  the  twelfth  current  version  Joint 
STARS  aircraft. 


results  of  this  mission  assessment  are  being  considered  in  the  Quadrennial  Defense  Review,  a 
process  in  which  DOD  is  taking  a  fundamental  look  at  its  strategy,  force  structure,  modernization, 
infrastructure,  and  readiness  to  meet  future  mission  requirements. 
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Production 
Commitment 
Unnecessary  and 
Risky 


We  have  previously  informed  dod  of  our  concerns  about  the  decision  to 
move  to  full-rate  production  in  spite  of  the  numerous  testing  deficiencies 
reported  by  both  afotec  and  dot&e.  dod  responded  that  in  making  the 
decision  to  move  to  full-rate  production,  it  “considered  the  test  reports 
(both  the  services’  and  the  Director,  Operational  Test  and  Evaluation’s), 
the  plans  to  address  the  deficiencies  identified  during  developmental  and 
operational  testing,  cost  estimates,  operational  requirements,  and  other 
program  information.” 

Although  DOD  believes  that  “none  of  the  deficiencies  identified  are  of  a 
scope  that  warrants  interrupting  production,”  the  production  decision 
memorandum  clearly  reflects  a  recognition  that  this  program  carries 
significant  risk.  In  his  memorandum,  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  directed  (1)  an  update  of  the  Joint  stars  Test 
and  Evaluation  Master  Plan  to  “address  multi-service  [operational  test  and 
evaluation]  deficiencies  (regression  testing);”  (2)  acceleration  of  the 
objective  and  threshold  dates  for  the  planned  Follow-on  Operational  Test 
and  Evaluation;  and  (3)  the  Air  Force  to  “fully  examine  [Joint  stars] 
affordability,  sustainability,  and  life  cycle  costs^®  including  the  scope  of 
contractor  use  for  field-level  system  support.” 

Factors  other  than  system  performance  may  have  influenced  the  decision 
to  move  Joint  stars  forward  into  fiill-rate  production,  dod’s  full-rate 
production  decision  for  this  program  occurred  during  the  same  time  frame 
the  Joint  stars  system  was  actively  being  promoted  as  the  U.S. 
government’s  candidate  for  meeting  NATO’s  military  requirement  for  a 
ground  surveillance  system.  For  example,  in  an  August  10,  1996, 
memorandum  to  the  Secretaries  of  State,  Defense,  and  Commerce  and  to 
the  Chairman  of  the  Joint  Chiefs  of  Staff,  the  Assistant  to  the  President  for 
National  Security  Affairs  stated  that  “We  have  been  working  through 
various  military,  diplomatic  and  poUtical  channels  to  secure  NATO  support 
for  a  Fall  1996  decision  in  principle  by  the  Conference  of  Armament 
Directors  ...  to  designate  [Joint  stars]  as  NATO’s  common  system.”  He 
went  on  to  state  that 


“I  am  writing  to  be  sure  you  know  that  the  President  is  personally  committed  to  [Joint 
stars],  has  engaged  Chancellor  Kohl  on  this  issue  and  will  continue  his  personal 
involvement  with  key  allies  to  ensure  our  goal  is  achieved.  I  would  ask  that  you  underscore 
your  personal  support  for  our  collective  efforts  on  behalf  of  [Joint  stars]  when  you  meet 
with  your  NATO  and  European  counterparts.” 


believe  this  analysis  should  include  examining  the  use  of  new  airframes. 
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Notwithstanding  dod’s  September  1996  conunitment  to  full-rate  Joint  stars 
production,  a  dod  official  informed  us  that  the  NATO  armament  directors  in 
their  November  1996  meeting  delayed  for  1  year  any  decision  on 
designating  Joint  stars  as  nato’s  common  system  or  pursuing  an  alternate 
system  to  be  developed. 


Follow-on  Operational 
Test  and  Evaluation 
Planned 


On  September  20, 1996,  dot&e  sent  to  Congress  a  Joint  stars  “Beyond  lrip” 
report  that  (1)  clearly  indicates  that  further  operational  testing  is  needed, 

(2)  could  only  declare  effectiveness  for  operations  other  than  war,  and 

(3)  stated  that  Joint  stars  is  imsuitable  as  tested.  On  September  25, 1996, 
DOD  approved  the  full-rate  production  of  Joint  stars.  In  the  acquisition 
memorandum  approving  Joint  stars  full-rate  production,  the  Under 
Secretary  of  Defense  for  Acquisition  and  Technology  called  for  an 
accelerated  foUow-on  operational  test  and  evaluation  of  Joint  stars  that  is 
to  address  the  deficiencies  identified  in  the  initial  operational  test  and 
evaluation  dot&e  reported  on  in  the  Beyond  lrip  report  to  Congress.  The 
planned  follow-on  operational  test  and  evaluation  will  provide  an 
opportunity  to  judge  the  Joint  stars  program’s  progress  in  resolving  the 
issues  identified  in  earlier  testing. 


In  the  process  of  moving  the  Joint  stars  program  forward  into  full-rate 
production,  dod  produced  a  Beyond  lrip  report  for  Congress  and  thus 
moved  past  a  key  congressional  reporting  requirement  that  serves  as  an 
important  risk  management  mechanism.  The  Beyond  lrip  report  to 
Congress  that  is  required  before  major  defense  acquisition  programs  can 
proceed  into  full-rate  production  serves  to  inform  Congress  of  the 
adequacy  of  the  operational  testing  done  on  the  system  and  to  provide  it 
with  a  determination  of  whether  the  system  has  demonstrated 
effectiveness  and  suitability.  Having  issued  this  report,  dot&e  is  under  no 
further  obligation  to  report  to  Congress  at  the  Beyond  lrip  report  level  of 
detail  on  the  adequacy  of  the  operational  testing  or  on  whether  the  system 
has  demonstrated  effectiveness  and  suitability  for  combat.  However,  dod 
plans  foUow-on  test  and  evaluation  of  the  system  to  address  the 
deficiencies  identified  during  the  system’s  earlier  testing. 


Conclusions 


Notwithstanding  any  concurrent  efforts  to  have  Joint  stars  designated  as  a 
NATO  common  system.  Joint  stars  test  performance  and  the  clearly 
unresolved  questions  about  its  operational  suitability  and  affordability 
should  have,  in  our  opinion,  caused  dod  to  delay  the  full-rate  production 
decision  vmtil  (1)  the  system  had,  through  the  planned  follow-on 
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operational  test  and  evaluation,  demonstrated  operational  effectiveness 
and  suitability;  (2)  the  Air  Force  had  completed  an  updated  analysis  of 
alternatives  for  the  Joint  stars  to  address  the  identified  aircraft  suitability 
and  cost  issues;  and  (3)  the  Air  Force  had  developed  an  analysis  to 
determine  whether  a  cost-effective  maintenance  concept  could  be 
designed  for  the  system.  Furthermore,  as  they  were  judged  “essential”  to 
mission  accomplishment  and  needed  “to  achieve  a  capable  Joint  stars 
system,”  the  satellite  communications  and  deployable  ground  support 
station  features  (present,  but  untested,  during  Operation  Joint  Endeavor) 
should  also  be  tested  during  the  planned  Joint  stars  follow-on  operational 
test  and  evaluation. 

Concerns  of  the  magnitude  discussed  in  this  report  are  not  indicative  of  a 
system  ready  for  full-rate  production.  The  program  should  have  continued 
under  lrip  until  the  issues  identified  by  afotec  and  dot&e  were  resolved 
and  the  system  was  shown  to  be  effective  and  suitable  for  combat. 
Furthermore,  the  recent  cost  growth  related  to  refurbishing  and  modifying 
the  old  airframes  being  used  for  Joint  stars  and  questions  regarding  the 
suitability  of  those  platforms  indicate  an  opportunity  to  reduce  the 
program’s  cost  and  improve  the  systems  acquired.  We  believe,  therefore, 
that  an  updated  study  of  the  cost  effectiveness  of  placing  Joint  stars  on 
new,  more  capable  aircraft  is  warranted. 


Recommendation 


We  recommend  that  the  Secretary  of  Defense  direct  the  Air  Force  to 
perform  an  analysis  of  possible  alternatives  to  the  current  Joint  stars  air 
platform,  to  include  placing  this  system  on  a  new  airframe. 


Matters  for 

Congressional 

Consideration 


Because  of  (1)  dod’s  decision  to  commit  to  fiill-rate  production  in  the  face 
of  the  test  results  discussed  in  this  report  and  (2)  its  subsequent  decision 
to  do  additional  tests  while  in  production  to  address  previous  test 
deficiencies,  we  are  convinced  that  dod  plans  to  proceed  with  the 
program.  However,  if  Congress  agrees  that  there  is  imnecessarily  high  risk 
in  this  program  and  believes  the  risk  should  be  reduced,  it  may  wish  to 
require  that: 

The  Air  Force  obtain  dot&e  approval  of  a  revised  test  and  evaluation 
master  plan  (and  all  plans  for  the  tests  called  for  in  that  master  plan)  for 
follow-on  operational  testing  to  include  adequate  coverage  of  gaps  left  by 
prior  testing  and  include  testing  of  any  added  features  considered  part  of 
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the  Standard  production  configuration  and  that  dot&e  considers  key 
system  components. 

•  DOT&E  provide  a  follow-on  test  and  evaluation  report  to  Congress 
evaluating  the  adequacy  of  all  testing  performed  to  judge  operational 
effectiveness  and  suitability  for  combat  and  a  definitive  statement  stating 
whether  the  system  has  demonstrated  operational  effectiveness  and 
suitability. 

•  DOD  develop  and  provide  Congress  an  analysis  of  alternatives  report  on  the 
Joint  STARS  air  platform  that  considers  the  suitability  of  the  current 
platform  and  other  cost-effective  alternatives,  and  the  Ufe-cycle  costs  of 
the  current  platform  and  best  alternatives. 


Agency  Comments 
and  Our  Evaluation 


In  commenting  on  a  draft  of  this  report,  dod  disagreed  with  our 
recommendation  that  the  Air  Force  be  directed  to  perform  an  analysis  of 
possible  alternatives  to  the  current  Joint  stars  air  platform.  It  also 
disagreed  with  our  suggestion  that  Congress  may  wish  to  require  dod  to 
develop  and  provide  Congress  a  report  on  that  analysis.  DOD  stated  that 
alternative  platforms  were  considered  prior  to  both  the  start  of  lrip  and 
the  fiill-rate  production  decision,  dod  stated  that  based  on  (1)  the  fact  that 
over  half  the  fleet  is  already  in  the  remanufacturing  process  or  delivered  to 
the  user;  (2)  the  large  nonrecurring  costs  that  would  be  associated  with 
moving  the  Joint  stars  mission  to  a  different  platform;  (3)  the  additional 
cost  to  operate  and  maintain  a  split  fleet  of  Joint  stars  airframes;  and 
(4)  the  expected  4-year  gap  in  deliveries,  such  a  strategy  would  force  the 
costs  of  moving  the  Joint  stars  mission  to  a  different  platform  outweigh 
the  benefits. 

dod’s  comment  about  having  previously  considered  alternative  platforms 
is  inconsistent  with  the  information  we  developed  during  our  review  and 
with  Air  Force  comments  provided  at  our  exit  conference.  In  an  effort  to 
reconcile  this  inconsistency,  we  requested  copies  of  the  prior  analyses  of 
alternative  platforms,  but  dod  was  not  able  to  provide  them,  dod’s 
statement  that  the  costs  of  moving  the  Joint  stars  mission  to  another 
platform  would  outweigh  the  benefits  contradicts  Command,  Control, 
Communications,  Computers,  Intelligence,  Surveillance,  and 
Reconnaissance  Mission  Assessment  briefings  given  the  Quadreimial 
Defense  Review.  Those  briefings  recormnend  (1)  limiting  Joint  stars 
production  to  12  aircraft,  (2)  moving  the  Joint  stars  mission  to  either 
corporate  jets  or  Unmaimed  Aerial  Vehicles,  and  (3)  phasing  out  Joint 
STARS  707  variants  as  quickly  as  the  new  platform  acquisitions  will  allow. 
According  to  those  briefings,  implementation  of  this  recorrunendation 
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could  result  in  a  more  effective  system  and  save  over  $3  billion  through 
fiscal  year  2010.  We  beheve  that  the  issue  clearly  warrants  further 
consideration.  Furthermore,  given  dod’s  resistance  to  the  concept,  we  are 
more  convinced  of  the  merits  of  our  suggestion  that  Congress  might  wish 
to  require  a  report  on  such  an  analysis. 

In  commenting  on  our  draft  report,  dod  also  indicated  that  congressional 
direction  was  xmneeded  on  our  suggestions  that  Congress  might  wish  to 
require  (1)  dot&e  approval  of  a  revised  test  and  evaluation  master  plan  for 
the  planned  Joint  stars  follow-on  operational  test  and  evaluation  and 
(2)  DOT&E  to  provide  Congress  with  a  follow-on  operational  test  and 
evaluation  report  on  the  adequacy  of  Joint  stars  testing  and  stating 
whether  Joint  stars  has  demonstrated  operational  effectiveness  and 
suitability,  dod  stated  that  congressional  direction  on  the  first  point  was 
unneeded  because  the  Joint  stars  full-rate  production  decision 
memorandum  required  that  the  test  and  evaluation  master  plan  be  updated 
for  Office  of  the  Secretary  of  Defense  approval  and  current  dod  policy  is 
that  DOT&E  will  review,  approve,  and  report  on  oversight  systems  in 
follow-on  operational  test  and  evaluation,  dod  also  stated  that 
congressional  direction  on  the  second  point  is  uimeeded  because  dot&e 
has  retained  Joint  stars  on  its  list  of  programs  for  oversight  and  is  to 
report  on  the  system  in  its  annual  report  to  Congress  as  appropriate. 

dod’s  response  did  not  directly  address  oxir  point  since,  as  dod  pointed  out, 
the  acquisition  decision  memorandum  that  approved  full-rate  production 
required  Office  of  the  Secretary  of  Defense  approval,  not  dot&e  approval, 
of  the  foUow-on  operational  test  and  evaluation  master  plan.  During  the 
course  of  our  review,  dod  officials  informed  us  that  there  was  significant 
disagreement  between  the  Air  Force  and  dot&e  as  to  what  follow-on 
testing  was  needed.  It  was  indicated  that  the  issue  would  probably  have  to 
be  resolved  at  higher  levels  within  the  department,  an  indication  of  greater 
flexibiUty  than  dod  imphes.  Furthermore,  while  dod  stated  there  were 
some  improvements  and  enhancements  “that  could  benefit  the  warfighter” 
and  acknowledged  that  those  features  were  not  tested,  it  did  not  respond 
to  our  comments  that  dot&e  judged  those  features  “essential”  to  mission 
accomplishment  or  commit  to  their  operational  test  and  evaluation.  Given 
these  facts,  we  have  not  only  maintained  our  suggestion  that  Congress 
may  wish  to  require  the  Air  Force  to  obtain  dot&e  approval  of  a  revised 
test  and  evaluation  master  plan,  but  also  strengthened  it  to  include  dot&e 
approval  of  supporting  test  plans. 
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In  its  response  to  our  suggestion  that  Congress  may  wish  to  require  that 
DOT&E  provide  it  a  detailed,  follow-on  test  and  evaluation  report,  dod  states 
congressional  direction  is  unnecessary  as  dot&e  will  report  on  the  system, 
among  many  others,  in  its  annual  report  to  Congress,  dod’s  comment  fails 
to  recognize,  however,  that  we  are  suggesting  that,  given  the  already 
reported  test  results.  Congress  may  wish  a  more  detailed  report  outlining 
the  adequacy  of  and  the  system’s  performance  during  follow-on 
operational  testing  to  help  in  its  oversight  and  provide  it  assurance  that 
the  system’s  problems  have  been  substantially  resolved.  Given  that 
(1)  Congress  felt  such  reporting  to  be  beneficial  enough  to  require  it 
before  a  system  can  proceed  beyond  lrip  and  (2)  the  fact  that  dot&e,  in  the 
required  report  provided  for  Joint  stars,  could  not  certify  effectiveness  for 
war  and  found  the  system  unsuitable  as  tested,  we  continue  to  believe  that 
Congress  may  wish  to  require  a  similar  report  based  on  the  follow-on 
operational  test  and  evaluation  planned. 

dod’s  comments  are  reprinted  in  their  entirety  in  appendix  I,  along  with 
our  evaluation. 


Scope  and 
Methodology 


To  determine  whether  Joint  stars  test  performance  indicates  a  maturity 
justifying  full-rate  production,  we  interviewed  officials  and  reviewed 
documents  in  Washington,  D.C.,  from  the  dod  Office  of  the  Director  of 
Operational  Test  and  Evaluation  and  the  Joint  stars  Integrated  Product 
Team.  We  reviewed  the  Air  Force  Operational  Test  and  Evaluation 
Center’s  multi-service  operational  test  and  evaluation  plan  and  its  final 
report  on  that  testing  and  the  dod  Director  of  Operational  Test  and 
Evaluation’s  Beyond  lrip  report.  To  determine  whether  dod  considered 
and  resolved  important  cost  and  performance  issues  prior  to  making  its 
full-rate  production  decision,  we  reviewed  Joint  stars  program  budget 
docmnents  and  program-related  memoranda  issued  by  the  Under 
Secretary  of  Defense  for  Acquisition  and  Technology.  To  determine 
whether  it  is  possible  that  a  more  useful  operational  test  and  evaluation 
report  can  be  provided  Congress,  we  reviewed  the  statute  governing 
operational  testing  and  evaluation,  examined  dot&e’s  Beyond  lrip  report, 
and  considered  other  relevant  program  information.  We  considered  and 
incorporated  where  appropriate  dod’s  response  to  our  September  20, 1996, 
letter  of  inquiry^^  and  its  response  to  a  draft  of  this  report.  We  conducted 
this  review  from  October  1996  through  April  1997  in  accordance  with 
generally  accepted  government  auditing  standards. 


Joint  STARS  Production  Decision  (GAO/NSIAD-96-242R,  Sept  20, 1996). 
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We  are  sending  copies  of  this  letter  to  other  appropriate  congressional 
committees;  the  Director,  Office  of  Management  and  Budget;  and  the 
Secretaries  of  Defense,  the  Army,  and  the  Air  Force.  Copies  will  also  be 
made  available  to  others  upon  request. 

If  you  or  your  staff  have  any  questions,  please  contact  me,  Mr.  Charles  F. 
Rey,  Assistant  Director,  or  Mr.  Bruce  Thomas,  Evaluator-in-Charge,  at 
(202)  5124841. 


Thomas  J.  Schulz 
Associate  Director, 

Defense  Acquisitions  Issues 
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List  of  Congressional  Committees 


The  Honorable  Strom  Thurmond 
Chairman 

The  Honorable  Carl  Levin 
Ranking  Minority  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Ted  Stevens 
Chairman 

The  Honorable  Daniel  K.  Inouye 
Ranking  Minority  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Floyd  D.  Spence 
Chairman 

The  Honorable  Ronald  V.  Dellums 
Ranking  Minority  Member 
Committee  on  National  Security 
House  of  Representatives 

The  Honorable  C.W.  Bill  Yoimg 
Chairman 

The  Honorable  John  P.  Murtha 
Ranking  Minority  Member 
Subcommittee  on  National  Security 
Committee  on  Appropriations 
House  of  Representatives 

The  Honorable  Richard  C.  Shelby 
Chairman 

The  Honorable  J.  Robert  Kerrey 
Vice  Chairman 

Select  Committee  on  Intelligence 
United  States  Senate 

The  Honorable  Porter  J.  Goss 
Chairman 

The  Honorable  Norman  D.  Dicks 
Ranking  Minority  Member 
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Select  Committee  on  Intelligence 
House  of  Representatives 
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Comments  From  the  Department  of  Defense 


Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


See  pp.  10-11  &  15-16. 


See  comment  1. 


Seepp.  10-11  &  15-16. 


ACQUIStTION  AND 
TECHNOLOGY 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-3000 


March  31 j  1997 


Mr.  Thomas  J.  Schulz 

Associate  Director,  Defense  Acquisition  Issues 
National  Security  and  International 
Affairs  Division 
U.S.  Genera!  Accounting  Office 
Washington,  DC  20548 


Dear  Mr.  Schulz, 


This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting  Office 
(GAO)  draft  report,  ‘TACTICAL  INTELLIGENCE:  Joint  STARS  Full-Rate  Production 
Decision  Premature  and  Risky,”  dated  February  14,  1997  (GAO  Code  707226),  OSD  Case  1296. 

The  DoD  nonconcurs  on  the  report  recommendation  that  directs  the  Air  Force  to  perform 
an  analysis  of  possible  alternatives  to  the  current  Joint  STARS  air  platform,  to  include  placing 
this  system  on  a  new  airframe. 

The  Joint  STARS  system  proceeded  into  full  rate  production  after  careful  consideration 
by  USD(A&T)  of  all  the  factors  affecting  the  program.  The  Services  are  fully  cognizant  of  areas 
where  deficiencies  exist,  both  in  effectiveness  and  suitability.  However,  these  deficiencies,  when 
taken  as  a  whole,  are  not  significant  enough  to  stop  production.  Deficiencies  not  withstanding, 
USCINCEUCOM  requested  and  received  Joint  STARS  support  from  a  second  deployment  of  the 
system  to  Operation  Joint  Endeavor  in  late  1996. 

Joint  STARS  will  continue  to  grow  over  time  into  a  fully  mature  system.  Given  that  the 
Initial  Operational  Capability  will  not  occur  until  September  1997,  the  users  are  satisfied  with  its 
performance  to  date.  Switching  to  an  alternate  airframe  at  this  point  in  the  program’s  life  would 
be  cost  prohibitive,  and  would  delay  giving  the  user  full  capability. 

The  detailed  DoD  comments  on  the  draft  report  are  provided  in  the  enclosure. 


Sincerely, 


( 


George  R.  Schneiter 
Director 

Strategic  and  Tactical  Systems 


Enclosure 


Page  22 


GAO/NSIAD-97-68  Joint  STARS  Production  Decision 


Appendix  I 

Comments  From  the  Department  of  Defense 


Now  on  p.  1. 
Now  on  p.  2. 


See  comment  2. 

Now  on  p.  2. 
Now  on  p.  3. 


See  comment  3. 


Now  on  p.  3. 


Detailed  Response  to 
GAO  Draft  Report 

“Joint  STARS  Full-Rate  Production  Decision  Premature  and  Risky” 


BACKGROUND  (Page  3,  GAO  Draft  Report) 

GAO  Comment,  Page  3,  Para  2:  “...AFOTEC  issued  its  Joint  STARS  multi-service 
operational  test  and  evaluation  final  report  on  June  14.  1996,  DOT&E  staff  analyzed  the  same 
data  and  the  Director  issued  his  **Bevond  LRIP”  report  to  Congress  on  September  20>  1996.” 

DoD  Response:  DOT&E  collected  and  analyzed  additional  data  beyond  that  provided  by 
AFOTEC.  These  data  included  interviews  with  ground  station  operators,  modeling  and 
simulation  to  support  required  terrain  analysis,  follow-up  data,  and  information  from  the 
Air  Force. 

RESULTS  IN  BRIEF  (Page  4,  GAO  Draft  Report) 

GAO  Comment,  Page  4,  Para  1:  “DoD's  decision  to  move  Joint  STARS  into  full-rate 
production  was  premature  and  raised  the  program  risks.” 

DoD  Response:  While  the  GAO  report  bases  most  of  its  concerns  on  data  contained  in 
the  Beyond  Low  Rate  Initial  Production  (BLRIP)  report,  this  report  was  only  one  factor 
considered  prior  to  the  favorable  Full-Rate  Production  decision  by  the  Defense  Acquisition 
Executive.  The  decision  to  move  Joint  STARS  into  full-rate  production  considered  the  Air  Force 
Operational  Test  and  Evaluation  Command’s  (AFOTEC)  Multi-service  Operational  Test  and 
Evaluation  (MOT&E)  report,  the  DOT&E*  s  BLRIP  report,  support  expressed  by  the  Joint  Chiefs 
of  Staff  and  the  operational  user,  and  the  assessment  of  the  overall  program  by  the  Overarching 
Integrated  Product  Team.  Additionally,  the  Services  presented  an  approach  and  committed  to 
correcting  shortcomings  surfaced  during  MOT&E.  The  Joint  STARS  acquisition  strategy  is 
based  on  procuring  two  E-8Cs  per  year  in  annual  lots;  the  same  rate  as  during  Low-Rate  Initial 
Production  (LRIP).  Hence,  the  risk  associated  with  significantly  increasing  the  production  rate  or 
committing  to  a  multi-year  buy  does  not  exist  on  the  Joint  STARS  program.  Many  of  the 
deficiencies  identified  during  operational  testing  will  be  corrected  by  software  updates,  readily 
incorporated  across  the  fleet. 

GAO  Comment,  Page  5,  Para  2:  “The  TDQT&EI  report  clearly  indicates  that  (1)  further 
operational  testing  is  needed.  (2)  DQT&E  could  only  declare  effectiveness  for  operations  other 
than  war,  and  (3)  the  system  was  unsuitable  as  tested.” 

DoD  Response:  (1)  The  Department  agrees  further  operational  testing  for  Joint 
STARS  is  necessary.  To  this  end,  the  Joint  STARS  Test  and  Evaluation  Master  Plan  (TEMP)  is 
being  revised. 


1 
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See  comment  4. 


Now  on  p.  4. 


Nowon  p.  4. 


See  comment  5. 

See  comment  6. 


(2)  In  contrast  to  DOT&E’s  evaluation,  the  Services  determined  Joint  STARS 
effective  during  MOT&E. 

(3)  DoD  response  to  the  suitability  concerns  are  addressed  in  the  GAO's  “Test  Results 
Do  Not  Support  Full-Rate  Production  (pages  12  and  13). 

GAO  Comment,  Page  6,  Para  1:  .  .it  may  now  be  more  cost  effective  for  the  Air  Force  to  buy 

that  [Boeing  767-2001  or  some  other  more  capable  aircraft.” 

DoD  Response:  The  DoD  addresses  this  subject  in  the  GAO*s  “Test  Results  Do  Not 
Support  Full-Rate  Production”  (page  6)  and  the  GAO  Recommendation  (page  21). 

TEST  RESULTS  DO  NOT  SUPPORT  FULL-RATE  PRODUCTION  (Page  6,  GAO  Draft 
Report) 

GAO  Comment,  Page  7,  Para  1:  “Joint  STARS  was  moved  from  low-rate  production  even 
though  (1)  it  performed  poorly  during  both  the  combined  development  test  and  operational  test 
and  the  operational  evaluation  in  Bosnia.  (2)  excessive  contractor  effort  was  needed  to  support 
Operation  Joint  Endeavor.  (3)  sustainability  of  the  system  is  questionable  since  the  system  is 
based  on  the  use  of  refurbished  25-30  year  old  airframes,  and  f4)  operational  software  is 
considered  significantly  immature.” 

DoD  Response:  Dr.  Kaminski,  Under  Secretary  of  Defense  (Acquisition  and 
Technology),  stated  in  his  September  1996  letter  to  Congress: 

“The  Joint  STARS  operational  evaluation  was  conducted  while  the  system  was 
supporting  the  Bosnia  Implementation  Force  in  Operation  Joint  Endeavor.  Evaluating  a 
weapon  system  under  contingency  operations  did  restrict  traditional  test  methodology  of 
using  controlled  scenarios,  but  afforded  us  an  opportunity  to  gain  invaluable  insight  under 
realistic  conditions.  Further,  it  saved  defense  resources  by  avoiding  a  separate  test  event. 
In  fact,  the  evaluation  involved  a  more  demanding,  complex  situation  than  the  original 
test  scenario.  During  Operation  Joint  Endeavor,  Joint  STARS  successfully  supported  100 
percent  of  its  assigned  operational  sorties  and  provided  command  and  control  and  battle 
management  capabilities  not  previously  available.  USCINCEUR  praised  Joint  STARS* 
performance.” 

“The  DOT&E  report  states  that  Joint  STARS  has  demonstrated  that  it  is  effective 
for  operations  other  than  war,  but  needs  several  added  features.  Most  of  the 
recommended  features  are  funded  as  part  of  the  baseline  program  or  identified  as  future 
system  upgrade  candidates.  Some  activities,  such  as  air  and  artillery  attack  support,  were 
not  needed  during  Joint  Endeavor.  Air  attacks  were  simulated  during  Joint  Endeavor  and 
had  been  demonstrated  during  Desert  Storm.  Future  exercises  and  tests  will  provide 
opportunity  for  further  demonstration  of  these  capabilities.** 
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See  comment  3. 


See  comment  7. 


See  comment  8. 


See  comment  8. 


The  Joint  STARS  system  demonstrated  capabilities  in  combined  developmental 
lest/operational  test  and  during  its  deployment  to  Operation  Joint  Endeavor  I  (OJE  I),  during 
which  MOT&E  was  conducted.  The  independent  testers,  system  users,  and  operational 
commanders  lauded  its  performance.  Joint  STARS  has  an  unprecedented  and  unmatched  record 
of  deploying  to  DESERT  STORM  and  Joint  ENDEAVOR,  operating  on  a  daily  basis,  and 
providing  138  sorties  to  the  theater  commanders  while  the  system  was  still  in  development.  The 
tests  found  deficiencies,  but  not  of  a  level  that  would  warrant  curtailing  production. 

(2)  The  level  of  participation  of  contractor  support  did  limit  the  extent  to  which  some 
testing  was  allowed  to  “count**;  however,  this  participation  was  deemed  necessary  to  support  the 
operational  mission.  The  level  of  contractor  support  during  MOT&E  was  not  unexpected,  due  to 
operational  requirements  for  support  to  the  CINC.  During  OJE  I,  one  of  the  two  deployed 
aircraft  was  an  £-8 A  development  model  that  by  contract  requires  contractor  support  and 
involvement  in  its  operation.  This  aircraft  was  not  used  to  collect  MOT&E  test  data.  The  Air 
Force  will  conduct  a  maintainability  demonstration  prior  to  declaration  of  Initial  Operational 
Capability  and  is  collecting  data  throughout  the  initial  fielding  of  the  system. 

The  most  telling  contractor-related  statistics  arise  when  comparing  OJE  I  (Dec  95- 
Mar  96)  contractor  support  versus  OJE  II  (Nov-Dec  96).  Contractor  involvement  during  OJE  I 
amounted  to  87  deployed  contractors  supporting  two  JSTARS  aircraft,  an  E-8A  (development 
model),  and  an  E-8C  (test  model).  OJE  11  deployment  consisted  of  one  E-8C  operational  aircraft 
and  one  E-8C  test  aircraft  which  was  eventually  replaced  by  a  second  E-8C  operational  aircraft. 
During  this  deployment  29  contractors  initially  supported  the  effort.  By  the  end  of  the 
deployment,  only  16  contractors  remained  deployed  and  none  of  these  individuals  was  on-board 
the  E-8C  aircraft  during  operational  missions.  This  reduction  of  support  was  the  result  of  using 
operational  aircraft,  improved  technical  orders,  trained  personnel,  and  added  infrastructure.  This 
represents  an  82%  reduction  in  contractor  support  when  comparing  OJE  I  and  OJE  H. 

(3)  In  September  1996,  Dr.  Kaminski,  USD(A&T),  stated  in  letters  sent  to  Members  of 
the  Senate  Armed  Services,  Senate  Appropriations,  and  House  Appropriations  Committees,  “The 
decision  to  refurbish  used  Boeing-707  aircraft  is  justified  since  the  system  meets  operational 
requirements.  In  addition,  moving  to  a  new  airframe  would  require  significant  development  and 
procurement  costs.  Airframe  and  power  plant  performance  and  maintenance  of  Joint  STARS  is 
similar  to  other  Boeing-707-type  aircraft  currently  in  the  Air  Force  inventory.” 

The  Draft  GAO  report  contends  that  the  cost  of  switching  from  a  Boeing  707  airframe  to 
a  Boeing  757  or  767  might  be  comparable  or  even  less  than  continuing  with  B-707s  for  all  19 
aircraft.  In  reaching  this  conclusion,  the  report  does  not  consider  all  related  procurement  costs  or 
additional  research  and  development  costs.  Alternative  platforms,  including  the  use  of  B- 
757/767  type  aircraft,  were  considered  prior  to  and  at  the  Low-Rate  Initial  Procurement 
milestone  decision  in  1993  and  reexamined  prior  to  the  1996  Full-Rate  Production  Decision. 

The  Boeing  707  meets  user  performance  requirements  and  offers  lower  life-cycle  costs  than 
converting  to  a  new  platform. 
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See  comment  8. 


See  comment  8. 


See  comment  4. 


Now  on  pp.  4-6. 


See  comment  3. 


As  staled  in  the  GAO  report,  procuring,  refurbishing,  and  modifying  used  B-707 
airframes  cost  approximately  $1(X)M.  A  new  767-200  would  cost  between  $82M  and  $93M  plus 
$40M  to  modify  the  aircraft  for  installation  of  Joint  STARS  mission  equipment.  Using  the  767 
AW  ACS  program  as  an  example,  the  conversion  from  a  707  to  767  extended  the  program  by  4 
years,  at  an  increased  cost  of  $451 M  (BY96$)  to  redesign  and  integrate  the  system. 

Additionally,  there  could  be  a  requirement  to  buy  a  767-based  test  asset,  estimated  at  $270M. 
Also,  the  767  cost  would  include  E-8C  Prime  Mission  Equipment  (PME)  installation.  Each 
“new”  767  would  also  require  up  to  $40M  in  modification  before  PME  installation.  Examples  of 
these  modifications  would  be  single-  point  aerial  refueling  capability  and  a  radar  hardback. 

The  current  cost  assessment  by  the  Air  Force  to  remanufacture,  modify,  and  configure  the 
final  eight  B-707s  is  $800M.  A  comparable  investment  in  eight  B-767s  would  be  $1,741M,  an 
increased  cost  of  $941 M. 

If  the  decision  were  made  to  support  a  “split”  fleet  of  767s  and  707s,  the  DoD  would  be 
confronted  with  higher  logistics  costs.  Initial  estimates  by  the  Joint  STARS  Depot  (W^amer 
Robins  Air  Logistics  Center)  indicate  a  rise  in  the  lifetime  Contractor  Logistics  Support  costs  of 
approximately  $180M.  This  would  be  for  additional  tooling,  parts,  and  training  required  for  the 
maintenance  of  two  separate  platforms. 

(4)  The  current  Joint  STARS  software  is  not  fully  mature  and  has  some  deficiencies. 

This  is  not  unusual  for  new,  complex  systems  such  as  Joint  STARS.  During  the  process  of 
preparing  for  operational  testing,  the  Air  Force  recognized  that  there  were  deficiencies  in  the 
software,  but  judged  it  mature  enough  for  MOT&E.  The  baseline  that  was  frozen  entering 
MOT&E  had  known  deficiencies  that,  based  on  user  priority,  will  be  fixed  in  the  pre-IOC  interim 
software  release  in  spring  1997  or  in  future  annual  software  releases.  That  judgment  was  borne 
out,  as  Joint  STARS  adequately  supported  the  operational  commander  during  OJE.  The  Joint 
STARS  system  is  still  over  six  months  away  from  IOC;  full  software  maturity  of  a  major  weapon 
system  is  not  a  requirement  for  IOC.  The  Air  Force  has  in  place  an  annual  release  program, 
beginning  this  year  with  a  pre-IOC  release,  to  handle  user  deficiencies.  Before  the  system  is 
declared  fully  operational  capable,  there  will  be  other  software  releases,  refining  its  capability 
further. 

An  engineering  and  manufacturing  development  effort  is  also  underway  to  replace  the 
Joint  STARS  General  Purpose  Computers.  This  upgrade  is  designed  to  expand  the  memory  and 
growth  capacity  of  the  system  as  well  as  solve  a  problem  involving  diminishing  manufacturing 
sources  for  the  present  equipment. 

GAO  Comment,  Page  7-9:  'Test  Results  disappointing.” 

DoD  Response:  The  DOT&E  BLRIP  and  the  Services*  MOT&E  reports  make 
contrasting  conclusions  regarding  Joint  STARS  performance  in  OJE  based  on  the  same  data. 

The  GAO  report  accurately  quotes  the  BLRIP  report;  however,  conclusions  by  the  agency  that 
executed  MOT&E  were  not  addressed.  The  Services  reported  that  the  Joint  STARS  was 
effective  and  suitable  with  deficiencies. 
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Now  on  p.  6. 


See  comment  9. 


Now  on  pp.  6-7. 


See  comment  10. 


Now  on  p.  7. 


See  comment  7. 


GAO  Comment,  Page  9,  Para  2:  “...DQT&E  also  noted  that  there  were  several  operational 
features  present  during  Joint  STARS  Operation  Joint  Endeavor  Deployment  that  were  essential 
to  its  mission  accomplishment  but  were  not  included  in  the  recent  production  decision.” 

DoD  Response:  DoD  agrees  that  there  were  some  improvements  and  enhancements 
employed  before  and  after  OJE  that  could  benefit  the  warfighter.  The  production  configuration 
had  been  already  baselined  without  these  enhancements  and  met  the  requirements  established  in 
the  Operational  Requirements  Document.  These  additional  capabilities  were  not  included  in  the 
production  version;  therefore,  they  were  not  tested.  These  enhancements  have  been  prioritized 
by  the  user,  and  many  are  now  funded  as  hardware  and  software  upgrades.  These  include 
Satellite  Communications  (SATCOM),  target  clutter  suppression,  and  communications 
improvements.  An  interim  contingency  package  including  SATCOM  and  a  deployable  ground 
support  station  has  been  put  together  to  give  the  users  many  of  these  capabilities  until  production 
equipment  or  upgrades  are  available. 

GAO  Comment,  Page  10-11:  ^^Significant  Contractor  Involvement/* 

DoD  Response:  The  contractors  participated  in  OJE  to  assist  the  4500  Joint  STARS 
Squadron  (JSS)  Commander  accomplish  the  mission.  During  OJE  I  and  the  MOT&E,  one  of  the 
two  deployed  aircraft  was  an  E-8A  development  model,  which,  because  it  had  not  been  released 
to  the  Government,  was  completely  supported  by  contractors.  This  aircraft  was  not  used  to 
collect  MOT&E  data.  Of  the  87  deployed  contractors  for  OJE  I,  12  provided  dedicated  E-8A 
airframe  support.  Another  17  provided  dedicated  support  of  deployed  equipment  not  part  of  the 
tested  baseline  and  that  were  not  delivery  items  to  the  government. 

Contractor  participation  did  limit  the  extent  to  which  some  testing  was  allowed  to  “count”; 
however,  it  was  deemed  necessary  to  support  the  operational  mission  of  OJE  I. 

Since  contractor  support  was  not  counted  in  the  MOT&E,  their  participation  in  OJE  had  no  impact 
on  test  results. 

Additional  information  regarding  contractor  support  of  Joint  STARS  during  OJE  operations 
is  in  response  to  GAO  draft  report  page  7,  para  1 . 

GAO  Comment,  Page  11,  Para  3:  .  .Significant  contractor  use  neither  supports  a  conclusion 

that  the  system  is  operationally  effective  or  suitable  for  combat,  nor  is  it  indicative  of  a  level  of 
system  maturity  that  justifies  full-rate  production.” 

DoD  Response:  Contractor  involvement  during  the  OJE  II  deployment  was  significantly 
smaller  than  OJE  I;  only  29  contractors  deployed  to  support  OJE  II.  Of  these,  17  provided 
aircraft  support,  and  5  of  those  were  dedicated  to  the  test  E-8C  aircraft.  By  the  end  of  the 
deployment,  only  16  contractors  remained  deployed;  none  flew  on-board  the  E-8Cs  during 
operational  missions.  This  represents  an  82%  reduction  in  contractor  support  from  OJE  I  to  OJE 
II.  OJE  II  demonstrated  that  the  “system”  is  maturing,  and  that  there  is  less  reliance  on 
contractor  support. 
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Nowon  p.  7. 


See  comment  11. 


See  comment  10. 


See  comment  6. 


Now  on  p.  8. 


See  comment  12. 


See  comment  1 1 . 


See  comment  12. 


GAO  Comment,  Page  12,  Para  1:  *^Joint  STARS  failure  to  meet  its  maintainability  criteria 
during  an  operation  less  demanding  than  combat,  even  with  such  significant  contractor 
involvement  beyond  that  planned  for  in  combat,  also  raises  the  question  of  the  Air  Force’s  ability 
to  develop  a  cost-effective  maintenance  plan  for  the  system.” 

DoD  Response:  The  original  planned  CONUS  MOT&E  would  have  been  more  limited 
than  the  deployed  operation,  particularly  in  the  areas  of  target  attack  support  and  battle 
management,  compared  to  the  opportunities  OJE  offered  to  judge  system  effectiveness  and 
suitability.  The  harsh  European  winter  was  demanding  on  flight  operations,  the  terrain  did  not 
enhance  radar  operations,  and  the  enemy  was  not  aligned  in  the  traditional  linear  battlefield. 
These  conditions  all  contributed  to  a  very  rigorous  test.  From  a  maintainability  perspective,  OJE 
I  was  a  very  demanding  environment.  Operations  were  maintained  24  hours  per  day  to  support 
an  average  12-hour  E-8A/E-8C  daily  sortie  length  for  nearly  100  days.  The  Desert  Storm  and 
MOT&E  environment  in  OJE  I  added  a  unique  and  realistic  environment  for  test. 

Gen.  George  Joulwan,  SACEUR,  stated  in  March  1996: 

“Your  [JSTARS]  efforts  in  support  of  US  and  Allied  troops  deployed  in 
the  former  Yugoslavia  have  been  greatly  appreciated  and  contributed 
immeasurably  to  the  safe  conduct  of  their  mission. .  ..Your  98%  mission 
ready  rate  and  100%  mission  completion  rate  are  the  envy  of  many  more 
mature  systems.’" 

GAO  Comment,  Page  12,  Para  2:  “System  Sustainability  and  suitability  is  questionable.” 

DoD  Response:  Although  concerns  were  raised  about  the  suitability  and  supportability 
of  the  E-8C,  the  user  is  satisfied  that  the  system  meets  requirements.  Performance,  reliability, 
and  maintainability  of  the  E-8C  are  similar  to  other  B-707-type  aircraft  in  the  Air  Force  inventory 
and  were  accepted  by  the  user  when  the  platform  was  selected  during  system  development  in  the 
late  1980’s,  and  at  the  low-rate  production  decision  in  1993.  The  projected  E-8C  operations 
tempo  and  the  ability  of  the  E-8C  airframe  to  sustain  operations  under  those  rates;  E-8C  aircraft 
engine  power  and  the  age  of  their  technology;  and  the  available  space/weight  for  growth  are  all 
matters  that  are  actively  monitored  and  managed  at  the  93^  Wing  and  Air  Logistics  Center  at 
Wamer-Robins  AFB. 

The  MOT&E  did  not  evaluate  the  operations  tempo,  so  test  data  do  not  exist  to 
realistically  challenge  the  E-8C*s  ability  to  maintain  the  required  tempo.  Sortie  Generation  Rate 
(SGR)  is  the  measurement  that  was  to  answer  this  issue;  however,  the  use  of  contractors 
impacted  this  measurement  and  precluded  its  non-biased  assessment.  The  planned  Regression 
Test  will  examine  this  measurement  and  at  that  time  operational  data  will  be  available  to 
determine  if  the  E-8C  can  meet  SGR  criteria. 

The  E-8C,  with  heavier  fuel  loads,  does  not  reach  the  upper  limits  of  the  operating 
altitude  until  adequate  fuel  has  been  consumed.  The  range  in  altitudes  is  selected  to  optimize  the 
performance  of  the  radar.  Not  attaining  the  maximum  operating  altitudes  until  well  into  the 
flight  has  little  impact  on  the  Joint  STARS  mission.  The  shortcoming  annotated  by  the  BLRIP  in 
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See  comment  6. 
Now  on  p.  8. 


See  comment  13. 


Now  on  p.  8. 


Now  on  p.  9. 


Now  on  p.  9. 


See  comment  14. 


terms  of  runway  length  required  and  fuel  weights  permitted  are  operational  limitations  of  this 
system.  The  operational  commanders  and  mission  planners  are  well  aware  of  these  conditions 
and  account  for  them.  These  conditions  were  fully  understood  in  OJE;  the  user  planned  for  them 
and  met  100%  of  the  assigned  missions. 

GAO  Comment,  Page  13,  Para  1;  **Another  area  of  Joint  STARS  suitability  is  the  svstem^s 
growth  potential.'' 

DoD  Response:  Currently  funded  preplanned  product  improvement  (P3I)  activities 
include  the  TADIL-J  upgrade.  Operations  and  Control  Program,  and  SATCOM.  The  SATCOM 
adds  six  components  and  300  pounds,  but  was  already  reflected  in  the  operating  weights  during 
OJE  I.  The  TADIL-J  upgrade  is  a  software-only  upgrade.  The  impact  of  the  Operations  and 
Control  Program  is  a  decrease  (approximately  2,000  pounds),  due  to  lower-weight  components, 
as  well  as  reducing  the  total  number  of  required  general  purpose  computers  from  five  to  two. 

GAO  Comment,  Page  13,  Para  2:  “DOT&E  also  noted  that  the  system’s  current  computers 
limited  its  growth  potential  due  to  their  having  very  little  reserve  processor  time  or  memory.” 

DoD  Response:  The  process  time  and  memory  reserve  for  the  central  processing  and 
general  processing  computers  are  being  addressed  by  commercial-off-the-shelf  technology 
(COTS)  improvements.  The  current  computers  resident  to  the  E-8C  are  scheduled  for 
replacement  with  COTS.  Upgrading  the  E-8C  computers  with  COTS  and  porting  software  to 
those  computers  will  reduce  the  number  of  computers  onboard  the  aircraft  and  increase  space, 
decrease  weight,  and  increase  reliability  while  simultaneously  reducing  processing  time  and 
expanding  memory  reserve. 

GAO  Comment,  Page  14,  Para  2:  “Operational  software  rated  significantly  immature.” 

DoD  Response:  At  the  conclusion  of  the  MOT&E,  the  using/developing  commands 
prioritized  deficiencies.  The  most  significant  operational  problems  were  immediately  addressed 
and  will  be  included  in  an  interim  software  update  with  a  planned  release  in  May  1997.  Since  the 
conclusion  of  MOT&E  (approximately  12  months  ago),  testing  of  the  interim  release  has  been 
underway.  The  interim  release  is  projected  to  correct  many  problems  documented  in  the 
MOT&E. 

GAO  Comment,  Page  15,  Para  2:  “How  easy  these  software  changes  arc  incorporated  remains 
to  be  seen  because  much  of  this  software,  according  to  AFOTEC  and  DOT&E.  is  poorly 
documented." 

DoD  Response:  Efforts  are  underway  to  improve  software  documentation.  Both 
DOT&E*s  and  AFOTEC’s  report  highlighted  these  problems.  Beginning  with  the  interim 
release,  better  documentation  is  being  provided  so  that  the  user  can  effectively  maintain  Joint 
STARS  software.  The  Air  Force  is  establishing  a  combined  organic/contractor  logistics  support 
software  maintenance  facility  at  Robins  AFB. 
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Nowon  p.  10. 


Nowon  p.  10. 

See  comment  8. 

Now  on  p.  12. 
Now  on  p.  12. 


See  comment  2. 
Now  on  p.  12. 


GAO  Comment,  Page  16,  Para  1:  “These  facts  in  combination  with  DoD’s  comments  raise  the 
serious  question  as  to  which  software  deficiencies  are  to  be  addressed  in  the  planned  April 
software  update.” 

DoD  Response:  Representatives  from  the  operational/developing  command  prioritized 
changes,  and  those  corrections  are  under  development. 

ALTERNATE  AIRCRAFT  SHOULD  BE  CONSIDERED  (Page  16,  GAO  Draft  Report) 

GAO  Comment,  Page  16,  Para  2:  “The  cost  of  procuring  and  preparing  new  aircraft  might  be 
comparable  or  even  less  than  the  current  cost." 

DoD  Response:  The  OSD  response  to  this  subject  is  addressed  in  the  GAO’s  ‘Test 
Results  Do  Not  Support  Full-Rate  Production  (page  7)  and  the  GAO  Recommendation  (page 
21)”. 

PRODUCTION  COMMITMENT  UNNECESSARY  AND  RISKY  (Page  17,  GAO  Draft 
Report) 

GAO  Comment,  Page  17,  Para  3;  ...in  making  the  decision  to  move  to  full-rate  production,  it 
“considered  the  test  reports  (both  the  Services'  and  the  Director.  Operational  Test  and 
Evaluation’s),  the  test  plans  to  address  the  deficiencies  identified  during  developmental  and 
operational  testing. . 

I>oD  Response:  This  statement  is  not  accurate.  The  quote,  which  is  extracted  from  a 
November  5,  1996,  OSD  letter  to  the  GAO,  states:  “...considered  the  lest  reports  (both  the 
Service’s  and  the  Director,  Operational  Test  and  Evaluation’s),  the  plans  to  address  deficiencies 
identified  during  developmental  and  operational  testing, . 

GAO  Comment,  Page  18,  Para  2:  “Although  DoD  believes  that  ‘none  of  the  deficiencies 
identified  are  of  a  scope  that  warrants  interrupting  production.’  the  production  decision 
memorandum  clearly  reflects  a  recognition  that  this  program  carries  significant  risk.”  In  his 
memorandum,  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology  directed  ( 1)  an 
update  of  the  Joint  STARS  Test  and  Evaluation  Master  Plan  to  “address  multi-Service 
foperational  test  and  evaluation!  deficiencies  (regression  testing),”  (2)  acceleration  of  the 
threshold  dates  for  the  planned  Follow-on  Test  and  Evaluation,  and  (3)  the  Air  Force  to  “fully 
examine  [Joint  STARS1  affordability,  sustainability,  and  life  cycle  costs  [this  should  include 
examining  the  use  of  new  air  frames),  including  the  scope  of  contractor  use  for  field-level  system 
support.” 

DoD  Response:  (1)  This  high-cost,  worldwide-deployable  system  requires  a  coordinated 
development/test  process  to  maximize  success.  The  TEMP  is  the  best  vehicle  to  accomplish  this. 
The  USD  (A&T)  has  directed  a  comprehensive  update  to  the  Joint  STARS  TEMP. 
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See  comment  15. 


See  comment  16. 


Now  on  p.  14. 


See  comment  8. 


Now  on  pp.  14-15. 


See  comments  1  and  17. 


(2)  FOT(&E  acceleration  does  not  reflect  a  recognition  of  risk,  but  an  alignment  of  testing 
with  the  delivery  of  Joint  STARS  upgrades. 

(3)  The  examination  of  affordability,  sustainability,  and  life  cycle  costs  recognizes  an 
effort  chartered  in  early  1996  by  the  Principal  Deputy  Under  Secretary  of  Defense  for  Acquisition 
and  Technology,  the  Air  Force  Service  Acquisition  Executive,  and  the  Joint  STARS  prime 
contractor  to  thoroughly  evaluate  and  control  program  costs.  The  direction  in  the  Acquisition 
Decision  Memorandum  served  to  have  the  results  of  the  study  (the  Skantze  study)  implemented 
and  funded. 

RECOMMENDATION  (Page  21,  GAO  Draft  Report) 

GAO  Comment,  Page  21,  Para  3:  We  recommend  that  the  Secretary  of  Defense  direct  the  Air 
Force  to  perform  an  analysis  of  possible  alternatives  to  the  current  Joint  STARS  air  platform,  to 
include  placing  this  system  on  a  new  airframe. 

DoD  Response:  Nonconcur.  Alternative  platforms  were  considered  prior  to  starting 
LRIP  and  prior  to  the  Full-Rate  Production  decision.  The  remanufactured  Boeing  707  was 
selected  as  meeting  user  performance  requirements  and  offering  lowest  program  life-cycle  cost. 
All  19  of  the  B-707  Joint  STARS  airframes  have  already  been  procured,  and  the  early 
procurement  of  these  aircraft  was  the  result  of  an  FY  1995  Congressional  initiative  to  pursue 
multi-airframe  procurement  to  reduce  the  acquisition  cost  of  the  program.  As  detailed  in  our 
response  to  the  sustainability  concern,  the  data  available  do  not  reflect  a  need  to  reinvestigate 
alternative  E-8C  platforms.  Based  on  (1)  Over  half  the  planned  fleet  already  is  either  in  the 
remanufacturing  process  or  delivered  to  the  user;  (2)  the  large  non-recurring  costs  associated 
with  rehosting  the  Joint  STARS  mission  on  a  different  platform;  (3)  the  additional  cost  to  operate 
and  maintain  a  split  fleet  of  Joint  STARS  airframes;  and  (4)  the  expected  four-year  gap  in 
deliveries  such  a  strategy  would  force,  the  costs  outweigh  the  benefits. 

MATTERS  FOR  CONGRESSIONAL  CONSIDERATION  (Page  22,  GAO  Draft  Report) 

GAO  Comment,  Page  22,  Para  2:  Congress  may  wish  to  require  that:  The  Air  Force  obtain 
DOT&E  approval  of  a  revised  test  and  evaluation  plan  for  follow-on  operational  testing  to 
include  adequate  coverage  of  gaps  left  bv  prior  testing  and  include  testing  of  any  added  features 
considered  part  of  the  standard  production  configuration  and  that  DOT&E  considers  key  system 
components. 

DoD  Response:  Partially  concur.  Congressional  direction  is  not  necessary,  as  the 
USD(A&T)’s  Acquisition  Decision  Memorandum  (ADM)  approving  Full-Rate  Production 
required  the  Air  Force  and  Army  to  update  the  Test  and  Evaluation  Master  Plan  for  OSD 
approval.  Current  Department  policy  specifies  that  DOT&E  will  review,  approve,  and  report  on 
oversight  systems  in  Follow-on  Operational  Test  and  Evaluation. 
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Now  on  p.  15. 

See  comments  1  and  18. 
Now  on  p.  15. 


See  comment  8. 


GAO  Comment,  Page  22,  Para  3:  DOT&E  provide  a  follow-on  test  and  evaluation  report  to 
Congress  evaluating  the  adequacy  of  all  testing  performed  to  judge  operational  effectiveness  and 
suitability  for  combat  and  a  definitive  statement  stating  whether  the  system  has  demonstrated 
operational  effectiveness  and  suitability. 

DoD  Response:  Partially  concur.  Congressional  direction  is  not  necessary,  as  DOT&E 
has  retained  Joint  STARS  on  their  list  of  programs  for  oversight  and  will  continue  to  report  on 
the  system  in  their  annual  report  to  Congress  and  other  reports  as  appropriate. 

GAP  Comment,  Page  22,  Para  4:  DoD  develop  and  provide  Congress  an  analysis  of 
alternatives  report  on  the  Joint  STARS  air  platform  that  considers  the  suitability  of  the  current 
platform  and  suitable  alternatives,  and  the  life-cycle  costs  of  the  current  platform  and  best 
alternatives. 


DoD  Response:  Nonconcur.  Alternative  platforms  were  considered  prior  to  starting 
LRIP  and  prior  to  the  Full-Rate  production  decision.  The  remanufactured  Boeing  707  was 
selected  as  meeting  user  performance  requirements  and  offering  lowest  program  life-cycle  cost. 
All  19  of  the  707  Joint  STARS  airframes  have  already  been  procured,  and  that  the  early 
procurement  of  these  aircraft  was  the  result  of  a  FY  1995  Congressional  initiative  to  pursue 
multi-airframe  procurement  to  reduce  the  acquisition  cost  of  the  program.  As  detailed  in  our 
response  to  the  sustainability  concern,  the  data  available  do  not  reflect  a  need  to  reinvestigate 
alternative  E-8C  platforms.  Based  on  (1)  Over  half  the  planned  fleet  already  is  either  in  the 
remanufacturing  process  or  delivered  to  the  user;  (2)  The  large  non-recurring  costs  associated 
with  rehosting  the  Joint  STARS  mission  on  a  different  platform;  (3)  The  additional  cost  to 
operate  and  maintain  a  split  fleet  of  Joint  STARS  airframes;  and  (4)  the  expected  four-year  gap 
in  deliveries  such  a  strategy  would  force,  the  costs  outweigh  the  benefits. 
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The  following  are  gao’s  comments  on  the  Department  of  Defense’s  (dod) 
letter  dated  March  31, 1997. 


GAO  Comments 


1.  We  have  not  suggested  or  recommended  that  Joint  stars  production  be 
interrupted.  We  have,  however,  suggested  actions  that  we  believe  (1)  will 
help  reduce  the  program’s  risk;  (2)  could  result  in  the  acquisition  of  a 
more  effective,  less  costly  system;  and  (3)  could  help  decisionmakers 
ensure  that  the  Joint  stapis  program  continues  to  make  progress. 

2.  The  report  has  been  modified  in  light  of  dod’s  comments. 

3.  dod’s  indication  that  other  factors  were  considered  in  deciding  to 
proceed  to  full-rate  production  is  a  signal  that  dod  and  the  Air  Force  are 
willing  to  accept  a  high  level  of  risk  even  when  the  Director,  Operation, 
Test,  and  Evaluation  (dot&e)  has  concluded  that  the  system  was 
unsuitable  as  tested  and  operational  effectiveness  for  war  remains  to  be 
demonstrated.  We  believe,  given  the  system’s  test  performance  as  reported 
by  both  the  Air  Force  Operational  Test  and  Evaluation  Command  (afotec) 
and  DOT&E  and  the  program’s  procurement  cost  growth  of  $1  billion 
between  the  low-rate  and  full-rate  production  decision  points,  that  an 
informed  full-rate  production  decision  required  the  following  information: 

(1)  an  approved  test  and  evaluation  master  plan  for  follow-on  operational 
testing  and  specific  plans  for  the  tests  called  for  in  that  master  plan, 

(2)  the  results  of  the  already  ongoing  study  of  ways  to  reduce  the 
program’s  cost,  and  (3)  an  analysis  of  alternatives  to  the  current  platform. 
DOD  did  not  have  these  items  in  hand  when  it  made  its  decision.  We  must 
also  note  that  dod  implies  that  our  reconunendations  would  require  a 
break  in  production.  This  is  inaccurate.  As  we  stated  in  the  body  of  our 
report,  the  program  could  have  continued  under  low-rate  initial 
production  (lrip)  until  operational  effectiveness  and  suitability  for  combat 
were  demonstrated  and  plans  to  address  identified  deficiencies  and  reduce 
program  costs  were  completed. 

4.  In  its  report  on  the  Joint  stars  multi-service  operational  test  and 
evaluation,  afotec  stated  that  “Joint  stars  software  is  immature  and 
significantly  impedes  the  system’s  reliabibty  and  effectiveness.”  We  do  not 
believe  that,  given  the  software  intensive  nature  of  the  system,  this 
statement  supports  a  conclusion  that  the  system  could  be  judged 
operationally  effective. 
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5.  We  must  note  that  follow-on  operational  test  and  evaluation  of  the 
system  was  planned  before  the  full-rate  production  decision.  The  full-rate 
production  decision  called  for  acceleration  of  that  testing  and  for  that 
testing  to  address  deficiencies  identified  in  the  earlier  tests.  Joint  stars 
could  have  continued  imder  lrip  pending  a  demonstration  of  operational 
effectiveness  and  suitability. 

6.  This  speaks  to  the  number  of  aircraft  missions  planned  and  the  number 
for  which  an  aircraft  was  provided.  It  does  not  address  the  quality  or 
quantity  of  the  support  provided  during  those  nussions.  Furthermore, 
dod’s  comment  refers  to  the  same — operation  that  is  reported  on  in  both 
the  Air  Force  and  dot&e  reports  and  in  this  report. 

7.  U.S.-based  contractor  support  was  utihzed  during  the  first  Operation 
Joint  Endeavor  deployment.  It  is  also  our  understanding  that  during  the 
second  Operation  Joint  Endeavor  deployment  the  Air  Force  may  have 
utilized  a  “reach-back”  maintenance  concept  in  which  U.S.  stationed 
contractor  staff  were  providing  field  support  through  satellite 
communications.  Moreover,  dod  and  Air  Force  officials  told  us  that  at  least 
at  the  beginrung  of  the  second  Operation  Joint  Endeavor  deployment, 
contractor  staff  were  flying  on  the  deployed  aircraft.  This  clearly  raises  the 
question  of  what  the  overall  level  of  contractor  support  was  for  both  the 
first  and  second  deployments. 

More  importantly,  a  decrease  in  the  level  of  contractor  support  between 
the  two  Operation  Joint  Endeavor  deployments  does  not  speak  to  (1)  the 
poor  test  results  during  the  first  deployment  with,  and  in  spite  of,  the  level 
of  contractor  support  or  (2)  the  quality  of  the  system’s  performance  during 
the  second  deployment;  that  is,  there  was  no  independent— dot&e— 
measurement  or  observation  of  how  the  system  performed  against  its 
operational  requirements. 

8.  Given  that  (1)  the  procurement  cost  growth  of  $760  million  for  19  Joint 
STARS  aircraft  since  the  low-rate  production  decision;  (2)  a  current 

707  airframe  purchase,  refurbishment,  and  modification  cost  of  about 
$110  milUon;  (3)  the  age  of  the  current  airframes — 25  to  30  years;  and 
(4)  the  $7  bilhon  estimated  operations  and  maintenance  life-cycle  cost  of 
those  aircraft,  we  continue  to  believe  that  an  analysis  of  alternatives  to  the 
current  air  platform  should  be  performed,  a  belief  bolstered  by  dod’s 
inability  to  provide  copies  of  its  reported  analyses  and  by  the 
recommendations  of  the  Command,  Control,  Communications, 

Computers,  Intelligence,  Surveillance,  and  Reconnaissance  Mission 
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Assessment,  as  discussed  on  pages  10, 11, 15,  and  16  of  this  report.  We 
believe  that  the  issue  clearly  warrants  further  study.  Furthermore,  given 
dod’s  resistance  to  the  concept,  we  are  more  convinced  of  the  merits  of 
our  suggestion  that  Congress  might  wish  to  require  a  report  on  such  an 
analysis. 

9.  DOD  stated  that  there  were  some  improvements  and  enhancements  “that 
could  benefit  the  warfighter”  but  acknowledges  that  those  features  were 
not  tested.  It  did  not  respond,  however,  to  our  comments  that  dot&e 
judged  some  of  those  features  “essential”  to  mission  accomplishment. 
Furthermore,  it  did  not  state  whether  those  features  would  be  subjected  to 
operational  testing  and  evaluation.  As  we  stated  in  the  body  of  our  report, 
since  at  least  two  of  the  features  present  during  Operation  Joint  Endeavor 
were  “essential”  to  its  mission  accomplishment,  have  already  been 
developed,  and  may  be  needed  “to  achieve  a  capable  Joint  stars  system,” 
those  features  should  also  be  tested  during  the  planned  Joint  stars 
follow-on  test  and  evaluation. 

10.  Given  the  level  of  contractor  support  during  the  multi-service 
operational  test  and  evaluation,  we  are  unable  to  understand  how  that 
support  could  not  have  impacted  upon  the  system’s  test  results.  As  we 
stated  in  our  report,  afotec  reported  that  80  contractors  were  deployed, 
“three  or  four  [contractor]  systems  engineers  flew  on  each  flight  to  ensure 
they  could  provide  system  stability  and  troubleshooting  expertise  during 
missions,”  and  “three  or  fom: . . .  were  on  the  ground  full  time,  researching 

and  developing  fixes  to  software  problems _ ”  Furthermore,  in  its  report 

DOT&E  states  that  “While  these  [contractor]  needs  show  that  [Joint  stars] 
requires  sophisticated  support,  they  may  also  mask  certain  deficiencies.”  It 
also  reported  that 


“As  already  discussed,  extensive  efforts  by  the  system  contractor  were  required  to  achieve 
the  demonstrated  availability  for  the  E-8C  aircraft.  Even  with  those  efforts  the  system  was 
not  able  to  meet  the  user  criteria  for  several  measures  directly  related  to  the  maintenance 
concept  in  place  during  [Operation  Joint  Endeavor] — a  concept  that  involved  considerably 
more  contractor  support  than  previously  envisioned,” 

11.  As  we  noted  in  the  body  of  our  report.  Joint  stars  failed  to  meet  test 
criteria  during  an  operation  less  demanding  than  combat,  even  with  such 
significant  contractor  involvement  beyond  that  planned  for  in  combat.  In 
discussing  operational  tempo  in  its  Beyond  lrip  report,  dot&e  stated  that  if 
the  system  is  operated  at  rates  similar  to  the  Airborne  Warning  and 
Control  System,  “it  is  questionable  whether  the  [Joint  stars  aircraft]  can 
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be  sustained  over  time.”  dod  commented  that  an  xmbiased  assessment  of 
the  measure  of  Joint  stars’  ability  to  maintain  the  required  tempo  could 
not  be  made  and  would  be  tested  during  the  follow-on  operational  test  and 
evaluation.  We  believe  that  an  informed  full-rate  production  decision 
requires  knowledge  of  a  system’s  ability  to  satisfy  the  operational  tempo 
expected  of  it.  dod  made  its  Joint  stars  full-rate  production  decision 
without  this  knowledge. 

12.  We  understand  that  Joint  stars,  like  most  systems,  has  limitations  that 
need  to  be  planned  arovmd.  At  issue  here  is  a  question  of  how  great  those 
limitations  are  and  whether  they  are  acceptable,  dod  states  that  “the  user 
is  satisfied  that  the  system  meets  requirements.”  However,  we  must  note 
that  the  Air  Force’s  own  Operational  Test  and  Evaluation  Center  reported 
that  the  “two  critical  suitability  [measures  of  performance,  sortie 
generation  rate  and  mission  reliability  rate],  were  affected  by  [Operation 
Joint  Endeavor]  contingency  requirements  and  system  stability  problems.” 

AFOTEC  stated  that  the  sortie  generation  rate  performance  was 
undetermined  and  judged  the  other  critical  suitability  measure  of 
performance — mission  reliability  rate — as  not  being  met.  In  discussing  the 
later  critical  measure  of  performance  afotec  reported  that 

“The  high  failure  rate  of  aging  aircraft  components  affected  [mission  reliability  rates]  as 
critical  failures  were  statistically  determined  to  affect  over  30  percent  of  the  sorties  flown. 
Analysis  revealed  the  elevated  critical  failure  rate  was  steady  and  showed  no  potential  for 
improvement.  Technical  data  and  software  immaturity  affected  the  maintainabihty  of  the 
aircraft,  and  contractor  involvement  further  compromised  clear  insight  into  the  Air  Force 
technicians’  ability  to  repair  the  system.” 


AFOTEC  also  reported  on  Joint  stars  performance  relative  to  15  supporting 
suitability  criteria.  It  stated  “Eight  did  not  meet  users’  criteria.  One  was  not 
tested.  Only  one  . . .  met  the  users’  criteria.  The  remaining  five  are  reported 
using  narrative  results.” 

13.  DOD  discusses  only  the  weight  growth  of  funded  activities,  leaving  open 
the  question  of  whether  there  are  future,  but  currently  unfunded, 
improvements  planned  that  will  add  weight  growth.  Air  Force  officials  told 
us  that  the  Airborne  Warning  and  Control  System  had  experienced  weight 
growth  over  the  life  of  its  program.  That  growth  was  attributed  to  the 
system’s  being  given  added  tasks  over  time.  We  believe  it  reasonable  to 
expect  that  the  Joint  stars  program  experience  might  track  that  of  the 
Airborne  Warning  and  Control  Systems  program,  that  is,  be  given  added 
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tasks  and  face  weight  growth  as  a  result.  Also,  regarding  Joint  stars  room 
for  groAvth,  dod  previously  advised  us  that  Joint  stars  currently  has  about 
455,000  cubic  inches  of  space  available.  We  must  note  that  this  equates  to 
a  volume  of  under  7  feet  cubed  and  that  in  commenting  on  the  system’s 
space  limitation,  dot&e  stated  “There  is  little  room  available  for  additional 
people  or  operator  workstations.” 

14.  As  we  stated  in  the  body  of  our  report,  how  easily  these  software 
changes  are  incorporated  remains  to  be  seen. 

15.  We  requested  and  dod  provided  additional  information  on  this  point. 
dod’s  subsequent  response  indicates  that  this  dod  comment  was  in  error. 

In  its  subsequent  response,  dod  stated  that  the  follow-on  test  and 
evaluation  was  accelerated  “to  reflect  [Office  of  the  Secretary  of  Defense] 
desire  for  earlier  [operational  test  and  evaluation]  to  evaluate  fixes  to 
[multi-service  operational  test  and  evaluation]  deficiencies.”  We  believe 
this  statement  reflects  a  recognition  of  increased  program  risk. 

16.  The  acquisition  decision  memorandum  approving  Joint  stars 
production  clearly  indicates  that  the  Skantze  study^  mentioned  was  not 
completed  at  that  time.  We  believe  that  the  full-rate  production  decision 
should  have  been  made  with  the  Skantze  study  in  hand.  Furthermore,  we 
do  not  vmderstand  why  dod  felt  the  need  to  direct  the  Air  Force  to  fund 
and  implement  a  plan  that  is  to  save  it  money,  but  felt  no  need  to  direct  the 
Air  Force  to  examine  alternative  platforms  that  at  least  one  other  dod 
panel  had  stated  would  not  only  save  $3  billion  but  also  provide  greater 
effectiveness. 

17.  We  believe  that  not  only  should  dot&e  approval  of  the  Joint  stars  Test 
and  Evaluation  Master  Plan  be  required,  but  also  of  all  supporting  test 
plans.  We  have  changed  the  language  of  this  matter  for  congressional 
consideration  accordingly. 

18.  We  are  suggesting  that  Congress  may  Avish  to  request  a  more  detailed 
report,  one  at  the  Beyond  lrip  report  level  of  detail,  a  level  of  detail  not 
provided  in  dot&e’s  annual  report.  Given  that  dot&e  could  only  state 
effectiveness  for  operations  other  than  war — could  not  state  a  belief  as  to 
whether  the  system  would  be  effective  in  two  of  the  three  critical 
operational  roles  it  is  expected  to  perform  in  war — and  found  the  system 
unsuitable  as  tested,  we  believe  that  such  report  would  help  Congress 


‘This  study  was  an  affordability  review  of  Joint  stars  that  examined  the  affordability,  sustainability, 
and  life-cycle  costs  to  thoroughly  evaluate  and  control  program  costs. 
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maintain  program  oversight,  dod’s  comment  of  “other  reports  as 
appropriate”  leaves  the  matter  in  dod’s  hand  to  decide  if  Congress  would 
benefit  from  such  a  report. 


(702226) 
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